Multi node Installation

saravanan30erd commented 3 years ago

Description

Until now this playbook supports only single node opensearch installation, I am adding the feature to support multi-node cluster installation.

Check List

[x] Commits are signed per the DCO using --signoff

saravanan30erd commented 3 years ago

/review @peterzhuamazon

peterzhuamazon commented 3 years ago

Hi @saravanan30erd this one probably will take some time as we are currently in the process of 1.2.0 release. I will probably start reviewing next week.

Thanks.

saravanan30erd commented 2 years ago

@peterzhuamazon any updates?

saravanan30erd commented 2 years ago

@peterzhuamazon any possibility to check in this week?

peterzhuamazon commented 2 years ago

Very sorry @saravanan30erd I spent most of my time on 1.2.0 and it finally gets out last week.

I will definitely test this week and could you look into #15 as well?

Again, very sorry for the delay and I will test and reply results.

Thanks.

peterzhuamazon commented 2 years ago

Interesting bug, os1 node I did not get a clean one, I just manually delete /usr/share/opensearch and it didnt recreate the directory and results in

fatal: [os1]: FAILED! => {"changed": false, "checksum": "910e2cbdc3648ada4609703321200d3a32223f17", "msg": "Destination directory /usr/share/opensearch/config does not exist"}

peterzhuamazon commented 2 years ago

I am not fully familiar with Ansible but is there a cleanup task you can create so people can safely uninstall older version of opensearch if they need to?

peterzhuamazon commented 2 years ago

Not able to start the process: errors_multinode_ansible.log

TASK [centos7/opensearch : Wait for opensearch to startup] *****************************************************************************************************************************************************************************************************************************************************************
fatal: [os2]: FAILED! => {"changed": false, "elapsed": 300, "msg": "Timeout when waiting for 0.0.0.0:9200"}
fatal: [os4]: FAILED! => {"changed": false, "elapsed": 300, "msg": "Timeout when waiting for 0.0.0.0:9200"}
fatal: [os3]: FAILED! => {"changed": false, "elapsed": 300, "msg": "Timeout when waiting for 0.0.0.0:9200"}
fatal: [os1]: FAILED! => {"changed": false, "elapsed": 300, "msg": "Timeout when waiting for 0.0.0.0:9200"}
fatal: [os5]: FAILED! => {"changed": false, "elapsed": 300, "msg": "Timeout when waiting for 0.0.0.0:9200"}

PLAY RECAP *****************************************************************************************************************************************************************************************************************************************************************************************************************
os1                        : ok=21   changed=11   unreachable=0    failed=1    skipped=13   rescued=0    ignored=0
os2                        : ok=16   changed=5    unreachable=0    failed=1    skipped=7    rescued=0    ignored=0
os3                        : ok=16   changed=5    unreachable=0    failed=1    skipped=7    rescued=0    ignored=0
os4                        : ok=16   changed=5    unreachable=0    failed=1    skipped=7    rescued=0    ignored=0
os5                        : ok=16   changed=5    unreachable=0    failed=1    skipped=7    rescued=0    ignored=0

Are you using opensearch user to start the process?

saravanan30erd commented 2 years ago

Interesting bug, os1 node I did not get a clean one, I just manually delete /usr/share/opensearch and it didnt recreate the directory and results

@peterzhuamazon Installation using RPM packages(via yum, etc..) will have proper cleanup steps. But here Opensearch doesn't have RPM, .deb packages yet. so I think we need to do proper cleanup (all the related files).

I am not fully familiar with Ansible but is there a cleanup task you can create so people can safely uninstall older version of opensearch if they need to?

Yeah, we can do this. I will create tasks for cleanup.

As of now, its better to do the installation on fresh nodes. I think we can have a call to close this PR. Let me know your availability so we can have a session.

peterzhuamazon commented 2 years ago

Interesting bug, os1 node I did not get a clean one, I just manually delete /usr/share/opensearch and it didnt recreate the directory and results

@peterzhuamazon Installation using RPM packages(via yum, etc..) will have proper cleanup steps. But here Opensearch doesn't have RPM, .deb packages yet. so I think we need to do proper cleanup (all the related files).

I am not fully familiar with Ansible but is there a cleanup task you can create so people can safely uninstall older version of opensearch if they need to?

Yeah, we can do this. I will create tasks for cleanup.

As of now, its better to do the installation on fresh nodes. I think we can have a call to close this PR. Let me know your availability so we can have a session.

Sent email. Thanks.

peterzhuamazon commented 2 years ago

After calling @saravanan30erd we are able to get it run

Example Logs

``` $ ansible-playbook -i inventories/opensearch/hosts opensearch.yml --extra-vars "admin_password=admin kibanaserver_password=admin" [WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details PLAY [opensearch installation & configuration] ***************************************************************************************************************************************************************************************************************************************************************************** TASK [centos7/opensearch : hostname] *************************************************************************************************************************************************************************************************************************************************************************************** ok: [os5] ok: [os4] ok: [os3] ok: [os1] ok: [os2] TASK [centos7/opensearch : Disable the selinux] **************************************************************************************************************************************************************************************************************************************************************************** [WARNING]: SELinux state change will take effect next reboot ok: [os2] ok: [os5] ok: [os1] ok: [os3] ok: [os4] TASK [centos7/opensearch : Hosts | populate inventory into hosts file] ***************************************************************************************************************************************************************************************************************************************************** ok: [os1] ok: [os3] ok: [os5] ok: [os4] ok: [os2] TASK [centos7/opensearch : Set vm.max_map_count in sysctl.conf] ************************************************************************************************************************************************************************************************************************************************************ ok: [os3] ok: [os5] ok: [os4] ok: [os1] ok: [os2] TASK [centos7/opensearch : Set open files limit in sysctl.conf] ************************************************************************************************************************************************************************************************************************************************************ ok: [os2] ok: [os3] ok: [os4] ok: [os5] ok: [os1] TASK [centos7/opensearch : OpenSearch Install | Download opensearch 1.1.0] ************************************************************************************************************************************************************************************************************************************************* ok: [os4] ok: [os2] ok: [os5] ok: [os3] ok: [os1] TASK [centos7/opensearch : OpenSearch Install | Create opensearch user] **************************************************************************************************************************************************************************************************************************************************** skipping: [os1] skipping: [os2] skipping: [os3] skipping: [os4] skipping: [os5] TASK [centos7/opensearch : OpenSearch Install | Create home directory] ***************************************************************************************************************************************************************************************************************************************************** skipping: [os1] skipping: [os2] skipping: [os3] skipping: [os4] skipping: [os5] TASK [centos7/opensearch : OpenSearch Install | Extract the tar file] ****************************************************************************************************************************************************************************************************************************************************** skipping: [os1] skipping: [os2] skipping: [os3] skipping: [os4] skipping: [os5] TASK [centos7/opensearch : OpenSearch Install | Copy Configuration File] *************************************************************************************************************************************************************************************************************************************************** changed: [os4] changed: [os5] changed: [os2] changed: [os3] changed: [os1] TASK [centos7/opensearch : OpenSearch Install | Copy jvm.options File for Instance] **************************************************************************************************************************************************************************************************************************************** ok: [os5] ok: [os4] ok: [os3] ok: [os2] ok: [os1] TASK [centos7/opensearch : OpenSearch Install | create systemd service] **************************************************************************************************************************************************************************************************************************************************** ok: [os1] ok: [os3] ok: [os2] ok: [os4] ok: [os5] TASK [centos7/opensearch : Security Plugin configuration | Create local temporary directory for certificates generation] *************************************************************************************************************************************************************************************************** changed: [os1 -> localhost] TASK [centos7/opensearch : Security Plugin configuration | Download certificates generation tool] ************************************************************************************************************************************************************************************************************************** changed: [os1 -> localhost] TASK [centos7/opensearch : Security Plugin configuration | Extract the certificates generation tool] *********************************************************************************************************************************************************************************************************************** changed: [os1 -> localhost] TASK [centos7/opensearch : Security Plugin configuration | Make the executable file] *************************************************************************************************************************************************************************************************************************************** ok: [os1 -> localhost] TASK [centos7/opensearch : Security Plugin configuration | Prepare the certificates generation template file] ************************************************************************************************************************************************************************************************************** changed: [os1 -> localhost] TASK [centos7/opensearch : Security Plugin configuration | Generate the node & admin certificates in local] **************************************************************************************************************************************************************************************************************** changed: [os1 -> localhost] TASK [centos7/opensearch : Security Plugin configuration | Copy the node & admin certificates to opensearch nodes] ********************************************************************************************************************************************************************************************************* changed: [os3] => (item=root-ca.pem) changed: [os2] => (item=root-ca.pem) changed: [os4] => (item=root-ca.pem) changed: [os5] => (item=root-ca.pem) changed: [os1] => (item=root-ca.pem) changed: [os3] => (item=root-ca.key) changed: [os2] => (item=root-ca.key) changed: [os4] => (item=root-ca.key) changed: [os5] => (item=root-ca.key) changed: [os1] => (item=root-ca.key) changed: [os3] => (item=os3.key) changed: [os4] => (item=os4.key) changed: [os2] => (item=os2.key) changed: [os5] => (item=os5.key) changed: [os1] => (item=os1.key) changed: [os3] => (item=os3.pem) changed: [os4] => (item=os4.pem) changed: [os2] => (item=os2.pem) changed: [os5] => (item=os5.pem) changed: [os1] => (item=os1.pem) changed: [os4] => (item=os4_http.key) changed: [os5] => (item=os5_http.key) changed: [os2] => (item=os2_http.key) changed: [os3] => (item=os3_http.key) changed: [os1] => (item=os1_http.key) changed: [os4] => (item=os4_http.pem) changed: [os5] => (item=os5_http.pem) changed: [os2] => (item=os2_http.pem) changed: [os3] => (item=os3_http.pem) changed: [os1] => (item=os1_http.pem) changed: [os4] => (item=admin.key) changed: [os5] => (item=admin.key) changed: [os2] => (item=admin.key) changed: [os3] => (item=admin.key) changed: [os1] => (item=admin.key) changed: [os4] => (item=admin.pem) changed: [os5] => (item=admin.pem) changed: [os2] => (item=admin.pem) changed: [os3] => (item=admin.pem) changed: [os1] => (item=admin.pem) TASK [centos7/opensearch : Security Plugin configuration | Copy the security configuration file 1 to cluster] ************************************************************************************************************************************************************************************************************** changed: [os2] changed: [os1] changed: [os4] changed: [os3] changed: [os5] TASK [centos7/opensearch : Security Plugin configuration | Copy the security configuration file 2 to cluster] ************************************************************************************************************************************************************************************************************** changed: [os3] changed: [os1] changed: [os2] changed: [os4] changed: [os5] TASK [centos7/opensearch : Security Plugin configuration | Prepare the opensearch security configuration file] ************************************************************************************************************************************************************************************************************* changed: [os1] changed: [os2] changed: [os3] changed: [os4] changed: [os5] TASK [centos7/opensearch : Security Plugin configuration | Set the file ownerships] **************************************************************************************************************************************************************************************************************************************** changed: [os3] changed: [os1] changed: [os2] changed: [os4] changed: [os5] TASK [centos7/opensearch : Security Plugin configuration | Set the folder permission] ************************************************************************************************************************************************************************************************************************************** ok: [os2] ok: [os1] ok: [os3] ok: [os4] ok: [os5] TASK [centos7/opensearch : Security Plugin configuration | Restart opensearch with security configuration] ***************************************************************************************************************************************************************************************************************** changed: [os4] changed: [os2] changed: [os5] changed: [os1] changed: [os3] TASK [centos7/opensearch : Pause for 3 seconds to provide sometime for OpenSearch start] *********************************************************************************************************************************************************************************************************************************** Pausing for 3 seconds (ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort) ok: [os1] TASK [centos7/opensearch : Security Plugin configuration | Copy the opensearch security internal users template] *********************************************************************************************************************************************************************************************************** changed: [os1] TASK [centos7/opensearch : Security Plugin configuration | Set the Admin user password] ************************************************************************************************************************************************************************************************************************************ changed: [os1] TASK [centos7/opensearch : Security Plugin configuration | Set the kibanaserver user pasword] ****************************************************************************************************************************************************************************************************************************** changed: [os1] TASK [centos7/opensearch : Security Plugin configuration | Initialize the opensearch security index in opensearch] ********************************************************************************************************************************************************************************************************* changed: [os1] TASK [centos7/opensearch : Remove `cluster.initial_master_nodes` setting from configuration] ******************************************************************************************************************************************************************************************************************************* changed: [os1] changed: [os2] changed: [os4] changed: [os3] changed: [os5] TASK [centos7/opensearch : Make sure opensearch is started] **************************************************************************************************************************************************************************************************************************************************************** ok: [os4] ok: [os5] ok: [os1] ok: [os2] ok: [os3] TASK [centos7/opensearch : Get all the installed ES plugins] *************************************************************************************************************************************************************************************************************************************************************** changed: [os4] changed: [os5] changed: [os1] changed: [os3] changed: [os2] TASK [centos7/opensearch : Show all the installed ES plugins] ************************************************************************************************************************************************************************************************************************************************************** ok: [os1] => { "msg": "opensearch-alerting\nopensearch-anomaly-detection\nopensearch-asynchronous-search\nopensearch-cross-cluster-replication\nopensearch-index-management\nopensearch-job-scheduler\nopensearch-knn\nopensearch-notebooks\nopensearch-performance-analyzer\nopensearch-reports-scheduler\nopensearch-security\nopensearch-sql" } ok: [os2] => { "msg": "opensearch-alerting\nopensearch-anomaly-detection\nopensearch-asynchronous-search\nopensearch-cross-cluster-replication\nopensearch-index-management\nopensearch-job-scheduler\nopensearch-knn\nopensearch-notebooks\nopensearch-performance-analyzer\nopensearch-reports-scheduler\nopensearch-security\nopensearch-sql" } ok: [os3] => { "msg": "opensearch-alerting\nopensearch-anomaly-detection\nopensearch-asynchronous-search\nopensearch-cross-cluster-replication\nopensearch-index-management\nopensearch-job-scheduler\nopensearch-knn\nopensearch-notebooks\nopensearch-performance-analyzer\nopensearch-reports-scheduler\nopensearch-security\nopensearch-sql" } ok: [os4] => { "msg": "opensearch-alerting\nopensearch-anomaly-detection\nopensearch-asynchronous-search\nopensearch-cross-cluster-replication\nopensearch-index-management\nopensearch-job-scheduler\nopensearch-knn\nopensearch-notebooks\nopensearch-performance-analyzer\nopensearch-reports-scheduler\nopensearch-security\nopensearch-sql" } ok: [os5] => { "msg": "opensearch-alerting\nopensearch-anomaly-detection\nopensearch-asynchronous-search\nopensearch-cross-cluster-replication\nopensearch-index-management\nopensearch-job-scheduler\nopensearch-knn\nopensearch-notebooks\nopensearch-performance-analyzer\nopensearch-reports-scheduler\nopensearch-security\nopensearch-sql" } TASK [centos7/opensearch : Wait for opensearch to startup] ***************************************************************************************************************************************************************************************************************************************************************** ok: [os4] ok: [os5] ok: [os1] ok: [os3] ok: [os2] TASK [centos7/opensearch : Check the opensearch status] ******************************************************************************************************************************************************************************************************************************************************************** changed: [os2] changed: [os1] changed: [os3] changed: [os5] changed: [os4] TASK [centos7/opensearch : Show the opensearch status] ********************************************************************************************************************************************************************************************************************************************************************* ok: [os3] => { "msg": { "active_primary_shards": 2, "active_shards": 7, "active_shards_percent_as_number": 100.0, "cluster_name": "development-cluster", "delayed_unassigned_shards": 0, "discovered_master": true, "initializing_shards": 0, "number_of_data_nodes": 5, "number_of_in_flight_fetch": 0, "number_of_nodes": 5, "number_of_pending_tasks": 0, "relocating_shards": 0, "status": "green", "task_max_waiting_in_queue_millis": 0, "timed_out": false, "unassigned_shards": 0 } } ok: [os2] => { "msg": { "active_primary_shards": 2, "active_shards": 7, "active_shards_percent_as_number": 100.0, "cluster_name": "development-cluster", "delayed_unassigned_shards": 0, "discovered_master": true, "initializing_shards": 0, "number_of_data_nodes": 5, "number_of_in_flight_fetch": 0, "number_of_nodes": 5, "number_of_pending_tasks": 0, "relocating_shards": 0, "status": "green", "task_max_waiting_in_queue_millis": 0, "timed_out": false, "unassigned_shards": 0 } } ok: [os1] => { "msg": { "active_primary_shards": 2, "active_shards": 7, "active_shards_percent_as_number": 100.0, "cluster_name": "development-cluster", "delayed_unassigned_shards": 0, "discovered_master": true, "initializing_shards": 0, "number_of_data_nodes": 5, "number_of_in_flight_fetch": 0, "number_of_nodes": 5, "number_of_pending_tasks": 0, "relocating_shards": 0, "status": "green", "task_max_waiting_in_queue_millis": 0, "timed_out": false, "unassigned_shards": 0 } } ok: [os5] => { "msg": { "active_primary_shards": 2, "active_shards": 7, "active_shards_percent_as_number": 100.0, "cluster_name": "development-cluster", "delayed_unassigned_shards": 0, "discovered_master": true, "initializing_shards": 0, "number_of_data_nodes": 5, "number_of_in_flight_fetch": 0, "number_of_nodes": 5, "number_of_pending_tasks": 0, "relocating_shards": 0, "status": "green", "task_max_waiting_in_queue_millis": 0, "timed_out": false, "unassigned_shards": 0 } } ok: [os4] => { "msg": { "active_primary_shards": 2, "active_shards": 7, "active_shards_percent_as_number": 100.0, "cluster_name": "development-cluster", "delayed_unassigned_shards": 0, "discovered_master": true, "initializing_shards": 0, "number_of_data_nodes": 5, "number_of_in_flight_fetch": 0, "number_of_nodes": 5, "number_of_pending_tasks": 0, "relocating_shards": 0, "status": "green", "task_max_waiting_in_queue_millis": 0, "timed_out": false, "unassigned_shards": 0 } } TASK [centos7/opensearch : Verify the roles of opensearch cluster nodes] *************************************************************************************************************************************************************************************************************************************************** changed: [os1] TASK [centos7/opensearch : Show the roles of opensearch cluster nodes] ***************************************************************************************************************************************************************************************************************************************************** ok: [os1] => { "msg": "ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name\n172.31.25.79 15 28 19 0.57 0.27 0.14 dm - os1\n172.31.29.187 28 28 12 0.46 0.23 0.14 dm * os2\n172.31.31.34 26 28 11 0.22 0.14 0.08 dm - os3\n172.31.18.22 8 28 10 0.71 0.34 0.17 di - os4\n172.31.22.134 11 29 11 0.35 0.20 0.14 di - os5" } PLAY [opensearch dashboards installation & configuration] ****************************************************************************************************************************************************************************************************************************************************************** TASK [centos7/dashboards : hostname] *************************************************************************************************************************************************************************************************************************************************************************************** changed: [dashboards1] TASK [centos7/dashboards : Disable the selinux] **************************************************************************************************************************************************************************************************************************************************************************** ok: [dashboards1] TASK [centos7/dashboards : Hosts | populate inventory into hosts file] ***************************************************************************************************************************************************************************************************************************************************** changed: [dashboards1] TASK [centos7/dashboards : Set open files limit in sysctl.conf] ************************************************************************************************************************************************************************************************************************************************************ ok: [dashboards1] TASK [centos7/dashboards : Dashboards Install | Download opensearch dashbaord 1.1.0] *************************************************************************************************************************************************************************************************************************************** changed: [dashboards1] TASK [centos7/dashboards : Dashboards Install | Create opensearch user] **************************************************************************************************************************************************************************************************************************************************** ok: [dashboards1] TASK [centos7/dashboards : Dashboards Install | Create home directory] ***************************************************************************************************************************************************************************************************************************************************** changed: [dashboards1] TASK [centos7/dashboards : Dashboards Install | Extract the tar file] ****************************************************************************************************************************************************************************************************************************************************** changed: [dashboards1] TASK [centos7/dashboards : Dashboards Install | Copy Configuration File] *************************************************************************************************************************************************************************************************************************************************** changed: [dashboards1] TASK [centos7/dashboards : Dashboards Install | create systemd service] **************************************************************************************************************************************************************************************************************************************************** changed: [dashboards1] TASK [centos7/dashboards : Make sure opensearch dashboards is started] ***************************************************************************************************************************************************************************************************************************************************** changed: [dashboards1] TASK [centos7/dashboards : Get all the installed dashboards plugins] ******************************************************************************************************************************************************************************************************************************************************* changed: [dashboards1] TASK [centos7/dashboards : Show all the installed dashboards plugins] ****************************************************************************************************************************************************************************************************************************************************** ok: [dashboards1] => { "msg": "alertingDashboards@1.1.0.0\nanomalyDetectionDashboards@1.1.0.0\nganttChartDashboards@1.1.0.0\nindexManagementDashboards@1.1.0.0\nnotebooksDashboards@1.1.0.0\nqueryWorkbenchDashboards@1.1.0.0\nreportsDashboards@1.1.0.0\nsecurityDashboards@1.1.0.0\ntraceAnalyticsDashboards@1.1.0.0" } PLAY RECAP ***************************************************************************************************************************************************************************************************************************************************************************************************************** dashboards1 : ok=13 changed=9 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 os1 : ok=36 changed=20 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 os2 : ok=23 changed=10 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 os3 : ok=23 changed=10 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 os4 : ok=23 changed=10 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 os5 : ok=23 changed=10 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 ```

Example Results

``` [root@os2 tmp]# curl https://172.31.29.187:9200/_cluster/health?pretty -u admin:admin --insecure { "cluster_name" : "development-cluster", "status" : "green", "timed_out" : false, "number_of_nodes" : 5, "number_of_data_nodes" : 5, "discovered_master" : true, "active_primary_shards" : 2, "active_shards" : 7, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 } [root@os2 tmp]# curl https://172.31.29.187:9200/_cat/nodes -u admin:admin --insecure 172.31.22.134 11 29 0 0.25 0.19 0.13 di - os5 172.31.25.79 15 30 3 0.41 0.25 0.14 dm - os1 172.31.18.22 8 28 0 0.51 0.32 0.17 di - os4 172.31.31.34 26 28 0 0.16 0.13 0.08 dm - os3 172.31.29.187 29 29 1 0.33 0.22 0.13 dm * os2 [root@os2 tmp]# curl https://172.31.29.187:9200/_cat/nodes?v -u admin:admin --insecure ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name 172.31.22.134 11 29 0 0.21 0.18 0.13 di - os5 172.31.25.79 16 33 19 0.51 0.28 0.15 dm - os1 172.31.18.22 8 28 0 0.43 0.31 0.17 di - os4 172.31.31.34 27 28 0 0.21 0.14 0.09 dm - os3 172.31.29.187 29 29 1 0.28 0.21 0.13 dm * os2 [root@os2 tmp]# curl https://172.31.29.187:9200/_cat/allocation?v -u admin:admin --insecure shards disk.indices disk.used disk.avail disk.total disk.percent host ip node 1 64.7kb 1.8gb 98.1gb 99.9gb 1 172.31.29.187 172.31.29.187 os2 1 64.7kb 1.8gb 98.1gb 99.9gb 1 172.31.18.22 172.31.18.22 os4 2 114.2kb 1.8gb 98.1gb 99.9gb 1 172.31.22.134 172.31.22.134 os5 2 114.2kb 4.1gb 95.8gb 99.9gb 4 172.31.25.79 172.31.25.79 os1 1 57.7kb 1.8gb 98.1gb 99.9gb 1 172.31.31.34 172.31.31.34 os3 [root@os2 tmp]# curl https://172.31.29.187:9200/_cat/indices?v -u admin:admin --insecure health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open security-auditlog-2021.12.02 Tu2PwBDxTDWXWnffEejiVQ 1 1 3 0 99kb 49.5kb green open .opendistro_security 35npPY8oRxmOlKPliL5kCQ 1 4 9 1 316.8kb 64.7kb ```

One issue is that on local machine where we run the ansible-playbook command will then create a temporary/tmp/opensearch-nodecerts folder to host all the certs and configs. Ansible will skip these copies if a previous old folder with the same name exist.

@saravanan30erd will send a new commit to fix it.

Thanks.

saravanan30erd commented 2 years ago

@peterzhuamazon pushed the fix for cleanup.

opensearch-project / ansible-playbook

Multi node Installation #13

Description

Check List