change searchEngine version to 0.5.0

[khaledk2]

I have updated searchengine-hosts.yml with the new searchengine version.

[sbesson]

After executing idr-searchengine.yml against prod114, I have the following running containers

[sbesson@prod114-searchengine ~]$ sudo docker ps
CONTAINER ID   IMAGE                                                  COMMAND                  CREATED              STATUS              PORTS                                            NAMES
c1a18db1db6a   openmicroscopy/omero-searchengine:0.5.0                "bash run_app.sh run…"   About a minute ago   Up About a minute   0.0.0.0:5577->5577/tcp, 8080/tcp                 searchengine
f0bbd0fab3a2   openmicroscopy/omero-searchengine:0.5.0                "bash run_app.sh cre…"   About a minute ago   Up About a minute   5577/tcp, 8080/tcp                               searchengine_create_index
15c94e7c9e1a   docker.elastic.co/elasticsearch/elasticsearch:7.16.2   "/bin/tini -- /usr/l…"   13 days ago          Up 13 days          0.0.0.0:9203->9200/tcp, 0.0.0.0:9303->9300/tcp   searchengine_elasticsearch_node3
2c5b3bc6821f   docker.elastic.co/elasticsearch/elasticsearch:7.16.2   "/bin/tini -- /usr/l…"   13 days ago          Up 13 days          0.0.0.0:9202->9200/tcp, 0.0.0.0:9302->9300/tcp   searchengine_elasticsearch_node2
974d31b25cd7   docker.elastic.co/elasticsearch/elasticsearch:7.16.2   "/bin/tini -- /usr/l…"   13 days ago          Up 13 days          0.0.0.0:9201->9200/tcp, 0.0.0.0:9301->9300/tcp   searchengine_elasticsearch_node1

Interestingly, an indexing process was spawned by the playbook which came as a bit as. a surprise to me. @khaledk2 @francesw I'll leave you to review if that's expected or if there are additional changes planned in the database before running the indexer.

[khaledk2]

@sbesson Sorry I should remind you about the issue that we had before, i.e. re-running the installation playbook will corrupt the elasticsearch data folder and make elasticsearch unusable. So, I have stopped all the containers and deleted the folder, could you please re-run the playbook again?

[sbesson]

So, I have stopped all the containers and deleted the folder, could you please re-run the playbook again?

Done.

[sbesson@prod114-searchengine ~]$ sudo docker ps
CONTAINER ID   IMAGE                                                  COMMAND                  CREATED          STATUS          PORTS                                            NAMES
ad2b92eb679e   openmicroscopy/omero-searchengine:0.5.0                "bash run_app.sh run…"   25 seconds ago   Up 22 seconds   0.0.0.0:5577->5577/tcp, 8080/tcp                 searchengine
636e547e86d0   docker.elastic.co/elasticsearch/elasticsearch:7.16.2   "/bin/tini -- /usr/l…"   53 seconds ago   Up 52 seconds   0.0.0.0:9203->9200/tcp, 0.0.0.0:9303->9300/tcp   searchengine_elasticsearch_node3
bb4f0429829d   docker.elastic.co/elasticsearch/elasticsearch:7.16.2   "/bin/tini -- /usr/l…"   55 seconds ago   Up 53 seconds   0.0.0.0:9202->9200/tcp, 0.0.0.0:9302->9300/tcp   searchengine_elasticsearch_node2
1fd3af6cd769   docker.elastic.co/elasticsearch/elasticsearch:7.16.2   "/bin/tini -- /usr/l…"   57 seconds ago   Up 55 seconds   0.0.0.0:9201->9200/tcp, 0.0.0.0:9301->9300/tcp   searchengine_elasticsearch_node1

Which exact commands should be run manually when deploying a new version of the search engine ? I'll capture this in the deployment troubleshooting section for now. If we keep working actively on the engine, this might be something that the playbook could handle eventually.

[khaledk2]

It may be a good idea to use a separate playbook to update the searchengine, so it will not touch the elasticsearch and its data folder, what do you think?

[khaledk2]

The following command will check the elastic search cluster health curl 127.0.0.1:9201/_cluster/health?pretty if everything is fine, as it is now, results should be something like that:

{
  "cluster_name" : "searchengine-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 8,
  "active_shards" : 22,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

If something is not correct, as it was before, you may get something like that:

  "error" : {
    "root_cause" : [
      {
        "type" : "master_not_discovered_exception",
        "reason" : null
      }
    ],
    "type" : "master_not_discovered_exception",
    "reason" : null
  },
  "status" : 503
}

[sbesson]

It may be a good idea to use a separate playbook to update the searchengine, so it will not touch the elasticsearch and its data folder, what do you think?

Do you means splitting idr-searchengine.yml into 2 sub-playbooks that would be called sequentially in https://github.com/IDR/deployment/blob/8356836dae5b5168b67259f6a1147c40ea591041/ansible/idr-02-services.yml#L13-L14? Happy with this approach if that creates a self-contained playbook that can be used to update the engine