search

IDR / deployment

Deployment infrastructure for the Image Data Resource
https://idr.openmicroscopy.org/about/deployment.html
BSD 2-Clause "Simplified" License
13 stars 14 forks source link

Mapr deprecation #441

Open will-moore opened 2 weeks ago

will-moore commented 2 weeks ago

This represents the changes I have been making manually on idr-testing with the replacement of mapr searches with searchengine, described at https://github.com/IDR/idr-gallery/pull/45

Summary of changes:

NB: I couldn't see where to add the file I created with:

sudo vi /usr/share/nginx/html/410.html (based on the 50x.html)

<!DOCTYPE html>
<html>
<head>
<title>Error</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<p>mapr/api pages have been removed.</p>
<p>
  Please use searchengine instead. See
  <a href="https://idr.openmicroscopy.org/searchengine/apidocs/">https://idr.openmicroscopy.org/searchengine/apidocs/</a>
</p>
</body>
</html>

Also the config for this file looks like:

    error_page 410 /410.html;
    location = /410.html {
        root   /usr/share/nginx/html;
    }

In my testing, I had added this in proxy-default.conf right before the location entry that I have added above:

    location ~ ^/mapr/api/(antibody|gene|compound|sirna|omap|phenotype|organism|orf|cellline|protein)($|/) {
        return 410;
    }

The error_page 410 about follows on from error_page 50x etc section in the same file, but I don't see that this is specified anywhere in this repo, so maybe that comes from existing nginx config somewhere?

will-moore commented 2 weeks ago

Build failures:

WARNING  Listing 3 violation(s) that are fatal
syntax-check[specific]: couldn't resolve module/action 'os_server_actions'. This often indicates a misspelling, missing collection, or incorrect module path.
decommission/archive-instance-networks.yml:35:5

syntax-check[specific]: The field 'hosts' has an invalid value, which includes an undefined variable. The error was: 'idr_environment' is undefined. 'idr_environment' is undefined
decommission/archive-instance-services.yml:3:3

syntax-check[specific]: couldn't resolve module/action 'letsencrypt'. This often indicates a misspelling, missing collection, or incorrect module path.
idr-proxy-letsencrypt.yml:43:5
sbesson commented 2 weeks ago

I believe the primary failure is actually during the converge step

  TASK [ome.iptables_raw : iptables-raw | enable iptables] ***********************
  Warning: : Collection community.docker does not support Ansible version 2.14.18
  Warning: : Collection community.docker does not support Ansible version 2.14.18
  Warning: : Collection community.docker does not support Ansible version 2.14.18
  fatal: [idr-omeroreadonly-docker]: FAILED! => {"changed": false, "msg": "Unable to start service iptables: Job for iptables.service failed because the control process exited with error code.\nSee \"systemctl status iptables.service\" and \"journalctl -xeu iptables.service\" for details.\n"}
  fatal: [idr-database-docker]: FAILED! => {"changed": false, "msg": "Unable to start service iptables: Job for iptables.service failed because the control process exited with error code.\nSee \"systemctl status iptables.service\" and \"journalctl -xeu iptables.service\" for details.\n"}
  fatal: [idr-omeroreadwrite-docker]: FAILED! => {"changed": false, "msg": "Unable to start service iptables: Job for iptables.service failed because the control process exited with error code.\nSee \"systemctl status iptables.service\" and \"journalctl -xeu iptables.service\" for details.\n"}

440 contains the proposal to address this issue. Re-expressing the next steps that need to happen:

1- https://github.com/ome/ansible-role-iptables-raw/pull/13 needs to be reviewed, merged and released 2- #440 needs to be updated with the new role version, reviewed and merged 3- origin/master can be merged into this branch to include the fix

will-moore commented 1 week ago

Seb: see https://github.com/ome/ansible-role-nginx-proxy/blob/1a2eb446951fe831126128c747beb6221d142047/templates/nginx-confd-proxy.j2#L50 for nginx setup.

"Could put 410.html under `about" or "static" locations. See https://github.com/IDR/idr.openmicroscopy.org/blob/master/omero-maintenance.html which does https://idr.openmicroscopy.org/about/omero-maintenance.html

will-moore commented 1 week ago

As discussed this morning in answer to my question above about where to put:

    error_page 410 /410.html;
    location = /410.html {
        root   /usr/share/nginx/html;
    }

It seems that the equivalent 50x handling is part of this ansible role:

https://github.com/ome/ansible-role-nginx-proxy/blob/1a2eb446951fe831126128c747beb6221d142047/templates/nginx-confd-proxy.j2#L50

However, it's not clear how/where to add the custom /usr/share/nginx/html/410.html I had above.

This is somewhat similar to the 'maintenance page' we have specified at https://github.com/IDR/idr.openmicroscopy.org/blob/master/omero-maintenance.html https://idr.openmicroscopy.org/about/omero-maintenance.html

However, to show a similar page for the /mapr/api/ urls would require a redirect, since we're returning a html page that is not from within nginx (as the 50x.html page is) but is hosted at a different URL.

So, the options are:

So, probably best to go for option 1. This may not be precisely the correct response, but it will be most user-friendly and easiest for us, bearing in mind it would be nice to release this next week or soon after.

will-moore commented 1 week ago

Build is green now 👍 . @dominikl will you be able to deploy this on a pilot so we can test? Then I can also deploy my idr-gallery changes on the same pilot and we can test with/without the idr-gallery changes. Thx

sbesson commented 1 week ago

@will-moore with #440 in, can you merge origin/master into your branch to fix the conflict?

will-moore commented 1 week ago

@sbesson Done

sbesson commented 1 week ago

Deployed to test125 together with #442 

TASK [ome.nginx_proxy : nginx | proxy cache config] ****************************************************************************************************************************
--- before: /etc/nginx/conf.d/proxy-cache.conf
+++ after: /Users/sbesson/.ansible/tmp/ansible-local-552739yr2e5xa/tmp3t9v16lg/nginx-confd-proxy-cache.j2
@@ -5,7 +5,6 @@
 proxy_cache_path /var/cache/nginx/omerorender levels=1:2 keys_zone=omerorender:35m max_size=40g inactive=180d use_temp_path=off;
 proxy_cache_path /var/cache/nginx/omerometadata levels=1:2 keys_zone=omerometadata:65m max_size=25g inactive=180d use_temp_path=off;
 proxy_cache_path /var/cache/nginx/omeroapi levels=1:2 keys_zone=omeroapi:320m max_size=10g inactive=180d use_temp_path=off;
-proxy_cache_path /var/cache/nginx/omeromapr levels=1:2 keys_zone=omeromapr:100m max_size=5g inactive=180d use_temp_path=off;
 proxy_cache_path /var/cache/nginx/omeroviewers levels=1:2 keys_zone=omeroviewers:1m max_size=1g inactive=180d use_temp_path=off;
 proxy_cache_path /var/cache/nginx/grafana levels=1:2 keys_zone=grafana:1m max_size=100m inactive=1m use_temp_path=off;

@@ -18,7 +17,6 @@
     "~web(client|gateway)/get_thumbnail*" 0;
     "~(webclient/)?api/*" 0;
     "~static/*" 0;
-    "~mapr/*" 0;
     "~gallery-api/*" 0;
     "~webclient/img_detail/*" 0;
     "~iviewer/*" 0;
@@ -46,7 +44,6 @@
     "~web(client|gateway)/render_*/*" omerorender;
     "~webclient/metadata_*" omerometadata;
     "~(webclient/)?api/*" omeroapi;
-    # "~mapr/*" omeromapr;
     "~webclient/img_detail/*" omeroviewers;
     "~iviewer/*" omeroviewers;
     "~grafana/*" grafana;

changed: [test125-proxy]

TASK [ome.nginx_proxy : nginx | proxy cache config] ****************************************************************************************************************************
skipping: [test125-proxy]

TASK [ome.nginx_proxy : nginx | check websockets configuration] ****************************************************************************************************************
skipping: [test125-proxy] => (item={'name': 'omerocached', 'location': '~ /webclient/metadata_*|/webclient/render_*|/webclient/get_thumbnail*|/webgateway/metadata_*|/webgateway/render_*|/webgateway/get_thumbnail*|/webclient/api/*|/webclient/search/*|/api/*|/webclient/img_detail/*|/iviewer/*|/figure/*|/gallery-api/*', 'server': 'http://omeroreadonly', 'cache_validity': '1d', 'maintenance_flag': '/srv/www/omero-maintenance.flag', 'maintenance_uri': '/about/omero-maintenance.html'}) 
skipping: [test125-proxy] => (item={'name': 'omeromapr', 'location': '~ /mapr/*', 'server': 'http://omeroreadonly', 'cache_validity': '180d', 'maintenance_flag': '/srv/www/omero-maintenance.flag', 'maintenance_uri': '/about/omero-maintenance.html'}) 
skipping: [test125-proxy] => (item={'name': 'omerostatic', 'location': '~ /static/*', 'server': 'http://omeroreadonly', 'cache_validity': '1d'}) 
skipping: [test125-proxy] => (item={'name': 'omero', 'location': '/', 'server': 'http://omeroreadonly', 'maintenance_flag': '/srv/www/omero-maintenance.flag', 'maintenance_uri': '/about/omero-maintenance.html'}) 
skipping: [test125-proxy] => (item={'name': 'omerowebsockets', 'location': '= /omero-ws', 'server': 'http://omeroreadonlywebsockets', 'websockets': True, 'websocketsonly': True, 'read_timeout': 86400}) 
skipping: [test125-proxy] => (item={'name': 'grafanarender', 'location': '^~ /grafana/', 'server': 'http://192.168.2.92:3000/render/dashboard/db/', 'cache_validity': '1m'}) 
skipping: [test125-proxy] => (item={'name': 'prometheusfederate', 'location': '^~ /prometheus/federate', 'server': 'http://192.168.2.92:9090/federate', 'cache_validity': '15s'}) 
skipping: [test125-proxy] => (item={'name': 'searchengine', 'location': '^~ /searchengine', 'server': 'http://192.168.2.197:5577/searchengine', 'host_header': '$host/searchengine'}) 

TASK [ome.nginx_proxy : nginx | proxy websockets config] ***********************************************************************************************************************
ok: [test125-proxy]

TASK [ome.nginx_proxy : nginx | proxy websockets config] ***********************************************************************************************************************
skipping: [test125-proxy]

TASK [ome.nginx_proxy : nginx | check redirect configuration] ******************************************************************************************************************
skipping: [test125-proxy]

TASK [ome.nginx_proxy : nginx | proxy redirects config] ************************************************************************************************************************
--- before: /etc/nginx/conf.d/proxy-redirect.conf
+++ after: /Users/sbesson/.ansible/tmp/ansible-local-552739yr2e5xa/tmpjl5hj5mc/nginx-confd-proxy-redirect.j2
@@ -14,8 +14,4 @@
     ~/pgpc /webclient/?show=screen-1151;
     /study/idr0083/figure/4i /webclient/img_detail/9822151/?dataset=10201&x=34105&y=84808&zm=25&c=1|318:9927$808080&m=g;
     /study/idr0083/figure/4r /webclient/img_detail/9822152/?dataset=10201&x=80560&y=77440&zm=66&c=1|495:9204$808080&m=g;
-    /mapr/api/antibody /searchengine/apidocs/;
-    /mapr/api/compound /searchengine/apidocs/;
-    /mapr/api/gene /searchengine/apidocs/;
-    /mapr/api/config /mapr/api/config;
 }

changed: [test125-proxy]

TASK [ome.nginx_proxy : nginx | proxy redirects config] ************************************************************************************************************************
skipping: [test125-proxy]

TASK [ome.nginx_proxy : nginx | stream-conf.d directory] ***********************************************************************************************************************
ok: [test125-proxy]

TASK [ome.nginx_proxy : nginx | stream proxy config] ***************************************************************************************************************************
skipping: [test125-proxy]

TASK [ome.nginx_proxy : nginx | stream proxy config] ***************************************************************************************************************************
ok: [test125-proxy]

TASK [ome.nginx_proxy : nginx | proxy config] **********************************************************************************************************************************
--- before: /etc/nginx/conf.d/proxy-default.conf
+++ after: /Users/sbesson/.ansible/tmp/ansible-local-552739yr2e5xa/tmpnatinmsu/nginx-confd-proxy.j2
@@ -12,7 +12,9 @@

     ssl_certificate /etc/ssl/localcerts/bundled.crt;
     ssl_certificate_key /etc/ssl/localcerts/server.key;
-    ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;  # don't use SSLv3 ref: POODLE
+    ssl_protocols  TLSv1.2 TLSv1.3;  # don't use SSLv3 ref: POODLE
+    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305;
+

     #charset koi8-r;
     #access_log  /var/log/nginx/log/host.access.log  main;
@@ -44,9 +46,8 @@
     location ~ ^/(mito|tara|pgpc|study)($|/) {
         return 302 $redirect_uri;
     }
-    
     location ~ ^/mapr/api/(?!config).*($|/) {
-        return 302 /searchengine/apidocs/;
+        return 302 $redirect_uri;
     }

     location = /webclient {

changed: [test125-proxy] => (item={'nginx_proxy_is_default': True, 'nginx_proxy_additional_directives': ['add_header Access-Control-Allow-Origin $allow_origin']})
--- before: /etc/nginx/conf.d/proxy-cachebuster.conf
+++ after: /Users/sbesson/.ansible/tmp/ansible-local-552739yr2e5xa/tmp2pvaxxro/nginx-confd-proxy.j2
@@ -33,6 +33,9 @@
     port_in_redirect off;

     location ~ ^/(mito|tara|pgpc|study)($|/) {
+        return 302 $redirect_uri;
+    }
+    location ~ ^/mapr/api/(?!config).*($|/) {
         return 302 $redirect_uri;
     }

changed: [test125-proxy] => (item={'nginx_proxy_server_name': 'cachebuster', 'nginx_proxy_listen_http': 0, 'nginx_proxy_ssl': False, 'nginx_proxy_cachebuster_enabled': True, 'nginx_proxy_backends': [{'name': 'omerocached', 'location': '~ /webclient/metadata_*|/webclient/render_*|/webclient/get_thumbnail*|/webgateway/metadata_*|/webgateway/render_*|/webgateway/get_thumbnail*|/webclient/api/*|/webclient/search/*|/api/*|/webclient/img_detail/*|/iviewer/*|/figure/*|/gallery-api/*|/mapr/*', 'server': 'http://omeroreadwrite', 'cache_validity': '1d', 'read_timeout': 900}, {'name': 'omerostatic', 'location': '~ /static/*', 'server': 'http://omeroreadwrite', 'cache_validity': '1d'}, {'name': 'omero', 'location': '/', 'server': 'http://omeroreadwrite'}]})
--- before: /etc/nginx/conf.d/proxy-idr-demo.openmicroscopy.org.conf
+++ after: /Users/sbesson/.ansible/tmp/ansible-local-552739yr2e5xa/tmpifpytgda/nginx-confd-proxy.j2
@@ -10,7 +10,9 @@

     ssl_certificate /etc/ssl/localcerts/bundled.crt;
     ssl_certificate_key /etc/ssl/localcerts/server.key;
-    ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;  # don't use SSLv3 ref: POODLE
+    ssl_protocols  TLSv1.2 TLSv1.3;  # don't use SSLv3 ref: POODLE
+    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305;
+

     #charset koi8-r;
     #access_log  /var/log/nginx/log/host.access.log  main;

changed: [test125-proxy] => (item={'nginx_proxy_server_name': 'idr-demo.openmicroscopy.org', 'nginx_proxy_ssl': True, 'nginx_proxy_redirect_map_locations': [], 'nginx_proxy_direct_locations': [{'location': '/', 'redirect301': '$scheme://idr.openmicroscopy.org$request_uri'}], 'nginx_proxy_backends': []})
will-moore commented 1 week ago

@sbesson - mostly that diff looks as expected (but not at

     }
-    
     location ~ ^/mapr/api/(?!config).*($|/) {
-        return 302 /searchengine/apidocs/;
+        return 302 $redirect_uri;
     }

I had manually added this myself by editing of /etc/nginx/conf.d/proxy-default.conf:

     location ~ ^/mapr/api/(?!config).*($|/) {
        return 302 /searchengine/apidocs/;
     }

so it looks like the diff is comparing my manual changes here to the changes after your update? With the change there return 302 $redirect_uri; this is failing to handle redirects e.g. for /mapr/api/gene/ because there's no entries for them in the proxy-redirect.conf.

But I also don't understand in proxy-redirect.conf we have this diff:

-    /mapr/api/antibody /searchengine/apidocs/;
-    /mapr/api/compound /searchengine/apidocs/;
-    /mapr/api/gene /searchengine/apidocs/;
-    /mapr/api/config /mapr/api/config;

This looks like removal of redirects to /searchengine/apidocs/. I didn't add these manually so I don't know where these came from?

sbesson commented 1 week ago

so it looks like the diff is comparing my manual changes here to the changes after your update?

That's correct, this is capturing the changes was applied to test125 from the state it was previously including local modifications.

This looks like removal of redirects to /searchengine/apidocs/. I didn't add these manually so I don't know where these came from?

Clearly someone edited this file as these changes are neither on prod124 or prod125.

If you don't want to use the redirect map, then you probably need to put your 302 redirect under nginx_proxy_direct_locations rather than nginx_proxy_redirect_map_locations

will-moore commented 1 week ago

Ah - understood! Done, thanks.

sbesson commented 1 week ago

After running the idr-proxy.yml playbook with the last commit

TASK [ome.nginx_proxy : nginx | proxy config] **************************************************************************************************************************************************************************************************************
--- before: /etc/nginx/conf.d/proxy-default.conf
+++ after: /Users/sbesson/.ansible/tmp/ansible-local-67117nlkwjgz4/tmp4odhrluu/nginx-confd-proxy.j2
@@ -46,9 +46,6 @@
     location ~ ^/(mito|tara|pgpc|study)($|/) {
         return 302 $redirect_uri;
     }
-    location ~ ^/mapr/api/(?!config).*($|/) {
-        return 302 $redirect_uri;
-    }

     location = /webclient {
         return 302 /webclient/?experimenter=-1;
@@ -76,6 +73,9 @@
     }
     location = /submitter-survey {
         return 302 https://forms.gle/beR4vqLrgLgRzgSX9;
+    }
+    location ~ ^/mapr/api/(?!config).*($|/) {
+        return 302 /searchengine/apidocs/;
     }

     location ~ /webclient/metadata_*|/webclient/render_*|/webclient/get_thumbnail*|/webgateway/metadata_*|/webgateway/render_*|/webgateway/get_thumbnail*|/webclient/api/*|/webclient/search/*|/api/*|/webclient/img_detail/*|/iviewer/*|/figure/*|/gallery-api/* {

changed: [test125-proxy] => (item={'nginx_proxy_is_default': True, 'nginx_proxy_additional_directives': ['add_header Access-Control-Allow-Origin $allow_origin']})
--- before: /etc/nginx/conf.d/proxy-cachebuster.conf
+++ after: /Users/sbesson/.ansible/tmp/ansible-local-67117nlkwjgz4/tmp4y0krw4k/nginx-confd-proxy.j2
@@ -35,9 +35,6 @@
     location ~ ^/(mito|tara|pgpc|study)($|/) {
         return 302 $redirect_uri;
     }
-    location ~ ^/mapr/api/(?!config).*($|/) {
-        return 302 $redirect_uri;
-    }

     location = /webclient {
         return 302 /webclient/?experimenter=-1;
@@ -65,6 +62,9 @@
     }
     location = /submitter-survey {
         return 302 https://forms.gle/beR4vqLrgLgRzgSX9;
+    }
+    location ~ ^/mapr/api/(?!config).*($|/) {
+        return 302 /searchengine/apidocs/;
     }

     location ~ /webclient/metadata_*|/webclient/render_*|/webclient/get_thumbnail*|/webgateway/metadata_*|/webgateway/render_*|/webgateway/get_thumbnail*|/webclient/api/*|/webclient/search/*|/api/*|/webclient/img_detail/*|/iviewer/*|/figure/*|/gallery-api/*|/mapr/* {

changed: [test125-proxy] => (item={'nginx_proxy_server_name': 'cachebuster', 'nginx_proxy_listen_http': 0, 'nginx_proxy_ssl': False, 'nginx_proxy_cachebuster_enabled': True, 'nginx_proxy_backends': [{'name': 'omerocached', 'location': '~ /webclient/metadata_*|/webclient/render_*|/webclient/get_thumbnail*|/webgateway/metadata_*|/webgateway/render_*|/webgateway/get_thumbnail*|/webclient/api/*|/webclient/search/*|/api/*|/webclient/img_detail/*|/iviewer/*|/figure/*|/gallery-api/*|/mapr/*', 'server': 'http://omeroreadwrite', 'cache_validity': '1d', 'read_timeout': 900}, {'name': 'omerostatic', 'location': '~ /static/*', 'server': 'http://omeroreadwrite', 'cache_validity': '1d'}, {'name': 'omero', 'location': '/', 'server': 'http://omeroreadwrite'}]})
ok: [test125-proxy] => (item={'nginx_proxy_server_name': 'idr-demo.openmicroscopy.org', 'nginx_proxy_ssl': True, 'nginx_proxy_redirect_map_locations': [], 'nginx_proxy_direct_locations': [{'location': '/', 'redirect301': '$scheme://idr.openmicroscopy.org$request_uri'}], 'nginx_proxy_backends': []})
will-moore commented 1 week ago

@sbesson Thanks - that looks like it's working fine now 👍 .

sbesson commented 1 week ago

@sbesson Thanks - that looks like it's working fine now 👍 .

👍 Do I understand correctly that this PR is now only waiting on a release of the idr-gallery application and the corresponding update to https://github.com/IDR/deployment/blob/cb12a873e37a1293418ae89ddeb10bf5ab4a01cf/ansible/group_vars/omero-hosts.yml#L245 ?

will-moore commented 1 week ago

Yes, I think so

will-moore commented 1 week ago

I'll wait for idr-gallery release (and mapr too) then bump them both in this PR.

will-moore commented 2 days ago

Testing on idr-testing looks good for me. Search, redirects etc.

francesw commented 2 days ago

Tested on idr-testing. Looks good. Search, redirects and pop-up working as expected.