Update to Elasticsearch 8 and secure the connection between searchengine and elasticsearch

[khaledk2]

This PR updates Elasticsearch to version 8 (8.8.1). It also secures the connection between the searchengine and the elasticsearch cluster and between the elasticsearch cluster nodes themselves. I have tested it locally and it worked fine. I am now testing it in pilot-idr0000-omeroreadwrite.
This PR required changes to the search engine code and I will create a PR for it soon.

[khaledk2]

I have created a seachengine PR (92) to support securing the connection between searchengine and Elasticsearch.

[khaledk2]

@sbesson I have created a docker image for the searchengine (khaledk2/searchengine:teste2_1) and used it to test the deployment.

[khaledk2]

I have passwords as private variables which have default values. This should work with management_tools 1692

[khaledk2]

Sorry, it seems that I have posted the following message to the wrong PR. The Elasticsearch nodes did not run correctly, I have checked that and found out that the CA file was not written to the mapped folder, so I have added a pause for 1 minute and tested that and it should work fine now. @sbesson I have stopped all the containers deleted the stopped containers and created folders, could you please re-run the playbook again?

[khaledk2]

I have renamed the elasticsearch_nodes vraible inside idr-searchengine.yml to elasticsearch_nodes_urlsas idr-elasticsearch.yml has a variable with the same name. When including the two playbooks in one playbook and running it, it will keep the items which have been added from the first included playbook inside the elasticsearch_nodes list when running the second included playbook rather than create a new list.

[khaledk2]

@sbesson Could you please delete the searchenginefolder in the /data before deployment in case you will use the same disk?

[sbesson]

@khaledk2 so far I have tested a complete recreation of the VM but if you are confident we can upgrade prod1119-searchengine without recreating the instance, that certainly would reduce the impact and the complexity. Minimally, we would 1- stop all the Docker instances, 2- delete /data before running the playbook. Anything else?

[khaledk2]

@sbesson, Yes, that would be fine, to be on the safe side, we may also delete all the stopped containers.

[sbesson]

Deployed on prod119

[sbesson@prod119-searchengine data]$ sudo docker ps
CONTAINER ID   IMAGE                                                 COMMAND                  CREATED          STATUS          PORTS                                            NAMES
18da47ac08f8   openmicroscopy/omero-searchengine:0.5.3               "bash run_app.sh run…"   25 seconds ago   Up 23 seconds   0.0.0.0:5577->5577/tcp, 8080/tcp                 searchengine
d2507ff519f5   docker.elastic.co/elasticsearch/elasticsearch:8.8.1   "/bin/tini -- /usr/l…"   2 minutes ago    Up 2 minutes    0.0.0.0:9203->9200/tcp, 0.0.0.0:9303->9300/tcp   searchengine_elasticsearch_node3
bcf010a6c401   docker.elastic.co/elasticsearch/elasticsearch:8.8.1   "/bin/tini -- /usr/l…"   2 minutes ago    Up 2 minutes    0.0.0.0:9202->9200/tcp, 0.0.0.0:9302->9300/tcp   searchengine_elasticsearch_node2
3e0fcc5dd9b0   docker.elastic.co/elasticsearch/elasticsearch:8.8.1   "/bin/tini -- /usr/l…"   2 minutes ago    Up 2 minutes    0.0.0.0:9201->9200/tcp, 0.0.0.0:9301->9300/tcp   searchengine_elasticsearch_node1

[khaledk2]

Looks good! thank you.