search

opensearch-project / ansible-playbook

🤖 A community repository for Ansible Playbook of OpenSearch Project.
https://opensearch.org/
Apache License 2.0
86 stars 96 forks source link

Support Amazon linux 2 #21

Closed saravanan30erd closed 2 years ago

saravanan30erd commented 2 years ago

Issues Resolved

17

@peterzhuamazon Tested the playbook with latest amazon linux 2, its compatible for almost all tasks except one which is fixed now.

Logs PLAY [opensearch installation & configuration] ************************************************************************************************************************ TASK [Gathering Facts] ************************************************************************************************************************************************ ok: [os3] ok: [os5] ok: [os4] ok: [os2] ok: [os1] TASK [centos7/opensearch : hostname] ********************************************************************************************************************************** changed: [os3] changed: [os5] changed: [os2] changed: [os4] changed: [os1] TASK [centos7/opensearch : Install the libselinux-python for Amazon Linux 2] ****************************************************************************************** changed: [os5] changed: [os3] changed: [os4] changed: [os1] changed: [os2] TASK [centos7/opensearch : Disable the selinux] *********************************************************************************************************************** ok: [os3] ok: [os4] ok: [os2] ok: [os5] ok: [os1] TASK [centos7/opensearch : Hosts | populate inventory into hosts file] ************************************************************************************************ changed: [os3] changed: [os4] changed: [os5] changed: [os2] changed: [os1] TASK [centos7/opensearch : Set vm.max_map_count in sysctl.conf] ******************************************************************************************************* changed: [os3] changed: [os5] changed: [os4] changed: [os2] changed: [os1] TASK [centos7/opensearch : Set open files limit in sysctl.conf] ******************************************************************************************************* changed: [os5] changed: [os3] changed: [os4] changed: [os2] changed: [os1] TASK [centos7/opensearch : OpenSearch Install | Download opensearch 1.2.0] ******************************************************************************************** changed: [os3] changed: [os4] changed: [os2] changed: [os1] changed: [os5] TASK [centos7/opensearch : OpenSearch Install | Create opensearch user] *********************************************************************************************** changed: [os5] changed: [os3] changed: [os1] changed: [os4] changed: [os2] TASK [centos7/opensearch : OpenSearch Install | Create home directory] ************************************************************************************************ changed: [os3] changed: [os5] changed: [os4] changed: [os2] changed: [os1] TASK [centos7/opensearch : OpenSearch Install | Extract the tar file] ************************************************************************************************* [WARNING]: Consider using the unarchive module rather than running tar. If you need to use command because unarchive is insufficient you can add warn=False to this command task or set command_warnings=False in ansible.cfg to get rid of this message. changed: [os5] changed: [os3] changed: [os4] changed: [os2] changed: [os1] TASK [centos7/opensearch : OpenSearch Install | Copy Configuration File] ********************************************************************************************** changed: [os3] changed: [os5] changed: [os4] changed: [os2] changed: [os1] TASK [centos7/opensearch : OpenSearch Install | Copy jvm.options File for Instance] *********************************************************************************** changed: [os3] changed: [os5] changed: [os4] changed: [os2] changed: [os1] TASK [centos7/opensearch : OpenSearch Install | create systemd service] *********************************************************************************************** changed: [os3] changed: [os5] changed: [os4] changed: [os2] changed: [os1] TASK [centos7/opensearch : Security Plugin configuration | Create local temporary directory for certificates generation] ********************************************** changed: [os1 -> localhost] TASK [centos7/opensearch : Security Plugin configuration | Download certificates generation tool] ********************************************************************* changed: [os1 -> localhost] TASK [centos7/opensearch : Security Plugin configuration | Extract the certificates generation tool] ****************************************************************** changed: [os1 -> localhost] TASK [centos7/opensearch : Security Plugin configuration | Make the executable file] ********************************************************************************** ok: [os1 -> localhost] TASK [centos7/opensearch : Security Plugin configuration | Prepare the certificates generation template file] ********************************************************* changed: [os1 -> localhost] TASK [centos7/opensearch : Security Plugin configuration | Generate the node & admin certificates in local] *********************************************************** changed: [os1 -> localhost] TASK [centos7/opensearch : Security Plugin configuration | Copy the node & admin certificates to opensearch nodes] **************************************************** changed: [os3] => (item=root-ca.pem) changed: [os5] => (item=root-ca.pem) changed: [os4] => (item=root-ca.pem) changed: [os3] => (item=root-ca.key) changed: [os1] => (item=root-ca.pem) changed: [os5] => (item=root-ca.key) changed: [os2] => (item=root-ca.pem) changed: [os3] => (item=os3.key) changed: [os5] => (item=os5.key) changed: [os4] => (item=root-ca.key) changed: [os3] => (item=os3.pem) changed: [os2] => (item=root-ca.key) changed: [os1] => (item=root-ca.key) changed: [os5] => (item=os5.pem) changed: [os3] => (item=os3_http.key) changed: [os4] => (item=os4.key) changed: [os5] => (item=os5_http.key) changed: [os3] => (item=os3_http.pem) changed: [os2] => (item=os2.key) changed: [os1] => (item=os1.key) changed: [os5] => (item=os5_http.pem) changed: [os3] => (item=admin.key) changed: [os4] => (item=os4.pem) changed: [os5] => (item=admin.key) changed: [os2] => (item=os2.pem) changed: [os3] => (item=admin.pem) changed: [os1] => (item=os1.pem) changed: [os5] => (item=admin.pem) changed: [os4] => (item=os4_http.key) changed: [os2] => (item=os2_http.key) changed: [os4] => (item=os4_http.pem) changed: [os1] => (item=os1_http.key) changed: [os2] => (item=os2_http.pem) changed: [os4] => (item=admin.key) changed: [os1] => (item=os1_http.pem) changed: [os2] => (item=admin.key) changed: [os4] => (item=admin.pem) changed: [os1] => (item=admin.key) changed: [os2] => (item=admin.pem) changed: [os1] => (item=admin.pem) TASK [centos7/opensearch : Security Plugin configuration | Copy the security configuration file 1 to cluster] ********************************************************* changed: [os5] changed: [os3] changed: [os2] changed: [os4] changed: [os1] TASK [centos7/opensearch : Security Plugin configuration | Copy the security configuration file 2 to cluster] ********************************************************* changed: [os5] changed: [os3] changed: [os4] changed: [os2] changed: [os1] TASK [centos7/opensearch : Security Plugin configuration | Prepare the opensearch security configuration file] ******************************************************** [WARNING]: Consider using the replace, lineinfile or template module rather than running sed. If you need to use command because replace, lineinfile or template is insufficient you can add warn=False to this command task or set command_warnings=False in ansible.cfg to get rid of this message. changed: [os3] changed: [os5] changed: [os2] changed: [os4] changed: [os1] TASK [centos7/opensearch : Security Plugin configuration | Set the file ownerships] *********************************************************************************** changed: [os5] changed: [os3] changed: [os4] changed: [os2] changed: [os1] TASK [centos7/opensearch : Security Plugin configuration | Set the folder permission] ********************************************************************************* changed: [os5] changed: [os3] changed: [os2] changed: [os4] changed: [os1] TASK [centos7/opensearch : Security Plugin configuration | Restart opensearch with security configuration] ************************************************************ changed: [os5] changed: [os3] changed: [os4] changed: [os2] changed: [os1] TASK [centos7/opensearch : Pause for 3 seconds to provide sometime for OpenSearch start] ****************************************************************************** Pausing for 3 seconds (ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort) ok: [os1] TASK [centos7/opensearch : Security Plugin configuration | Copy the opensearch security internal users template] ****************************************************** changed: [os1] TASK [centos7/opensearch : Security Plugin configuration | Set the Admin user password] ******************************************************************************* changed: [os1] TASK [centos7/opensearch : Security Plugin configuration | Set the kibanaserver user pasword] ************************************************************************* changed: [os1] TASK [centos7/opensearch : Security Plugin configuration | Initialize the opensearch security index in opensearch] **************************************************** changed: [os1] TASK [centos7/opensearch : Security Plugin configuration | Cleanup local temporary directory] ************************************************************************* changed: [os1 -> localhost] TASK [centos7/opensearch : Remove `cluster.initial_master_nodes` setting from configuration] ************************************************************************** changed: [os3] changed: [os5] changed: [os2] changed: [os4] changed: [os1] TASK [centos7/opensearch : Make sure opensearch is started] *********************************************************************************************************** ok: [os3] ok: [os5] ok: [os4] ok: [os2] ok: [os1] TASK [centos7/opensearch : Get all the installed ES plugins] ********************************************************************************************************** changed: [os5] changed: [os3] changed: [os4] changed: [os2] changed: [os1] TASK [centos7/opensearch : Show all the installed ES plugins] ********************************************************************************************************* ok: [os1] => { "msg": "opensearch-alerting\nopensearch-anomaly-detection\nopensearch-asynchronous-search\nopensearch-cross-cluster-replication\nopensearch-index-management\nopensearch-job-scheduler\nopensearch-knn\nopensearch-observability\nopensearch-performance-analyzer\nopensearch-reports-scheduler\nopensearch-security\nopensearch-sql" } ok: [os2] => { "msg": "opensearch-alerting\nopensearch-anomaly-detection\nopensearch-asynchronous-search\nopensearch-cross-cluster-replication\nopensearch-index-management\nopensearch-job-scheduler\nopensearch-knn\nopensearch-observability\nopensearch-performance-analyzer\nopensearch-reports-scheduler\nopensearch-security\nopensearch-sql" } ok: [os3] => { "msg": "opensearch-alerting\nopensearch-anomaly-detection\nopensearch-asynchronous-search\nopensearch-cross-cluster-replication\nopensearch-index-management\nopensearch-job-scheduler\nopensearch-knn\nopensearch-observability\nopensearch-performance-analyzer\nopensearch-reports-scheduler\nopensearch-security\nopensearch-sql" } ok: [os4] => { "msg": "opensearch-alerting\nopensearch-anomaly-detection\nopensearch-asynchronous-search\nopensearch-cross-cluster-replication\nopensearch-index-management\nopensearch-job-scheduler\nopensearch-knn\nopensearch-observability\nopensearch-performance-analyzer\nopensearch-reports-scheduler\nopensearch-security\nopensearch-sql" } ok: [os5] => { "msg": "opensearch-alerting\nopensearch-anomaly-detection\nopensearch-asynchronous-search\nopensearch-cross-cluster-replication\nopensearch-index-management\nopensearch-job-scheduler\nopensearch-knn\nopensearch-observability\nopensearch-performance-analyzer\nopensearch-reports-scheduler\nopensearch-security\nopensearch-sql" } TASK [centos7/opensearch : Wait for opensearch to startup] ************************************************************************************************************ ok: [os5] ok: [os3] ok: [os4] ok: [os2] ok: [os1] TASK [centos7/opensearch : Check the opensearch status] *************************************************************************************************************** [WARNING]: Consider using the get_url or uri module rather than running curl. If you need to use command because get_url or uri is insufficient you can add warn=False to this command task or set command_warnings=False in ansible.cfg to get rid of this message. changed: [os3] changed: [os2] changed: [os5] changed: [os4] changed: [os1] TASK [centos7/opensearch : Show the opensearch status] **************************************************************************************************************** ok: [os1] => { "failed_when_result": false, "msg": { "active_primary_shards": 2, "active_shards": 7, "active_shards_percent_as_number": 100.0, "cluster_name": "development-cluster", "delayed_unassigned_shards": 0, "discovered_master": true, "initializing_shards": 0, "number_of_data_nodes": 5, "number_of_in_flight_fetch": 0, "number_of_nodes": 5, "number_of_pending_tasks": 0, "relocating_shards": 0, "status": "green", "task_max_waiting_in_queue_millis": 0, "timed_out": false, "unassigned_shards": 0 } } ok: [os2] => { "failed_when_result": false, "msg": { "active_primary_shards": 2, "active_shards": 7, "active_shards_percent_as_number": 100.0, "cluster_name": "development-cluster", "delayed_unassigned_shards": 0, "discovered_master": true, "initializing_shards": 0, "number_of_data_nodes": 5, "number_of_in_flight_fetch": 0, "number_of_nodes": 5, "number_of_pending_tasks": 0, "relocating_shards": 0, "status": "green", "task_max_waiting_in_queue_millis": 0, "timed_out": false, "unassigned_shards": 0 } } ok: [os3] => { "failed_when_result": false, "msg": { "active_primary_shards": 2, "active_shards": 7, "active_shards_percent_as_number": 100.0, "cluster_name": "development-cluster", "delayed_unassigned_shards": 0, "discovered_master": true, "initializing_shards": 0, "number_of_data_nodes": 5, "number_of_in_flight_fetch": 0, "number_of_nodes": 5, "number_of_pending_tasks": 0, "relocating_shards": 0, "status": "green", "task_max_waiting_in_queue_millis": 0, "timed_out": false, "unassigned_shards": 0 } } ok: [os4] => { "failed_when_result": false, "msg": { "active_primary_shards": 2, "active_shards": 7, "active_shards_percent_as_number": 100.0, "cluster_name": "development-cluster", "delayed_unassigned_shards": 0, "discovered_master": true, "initializing_shards": 0, "number_of_data_nodes": 5, "number_of_in_flight_fetch": 0, "number_of_nodes": 5, "number_of_pending_tasks": 0, "relocating_shards": 0, "status": "green", "task_max_waiting_in_queue_millis": 0, "timed_out": false, "unassigned_shards": 0 } } ok: [os5] => { "failed_when_result": false, "msg": { "active_primary_shards": 2, "active_shards": 7, "active_shards_percent_as_number": 100.0, "cluster_name": "development-cluster", "delayed_unassigned_shards": 0, "discovered_master": true, "initializing_shards": 0, "number_of_data_nodes": 5, "number_of_in_flight_fetch": 0, "number_of_nodes": 5, "number_of_pending_tasks": 0, "relocating_shards": 0, "status": "green", "task_max_waiting_in_queue_millis": 0, "timed_out": false, "unassigned_shards": 0 } } TASK [centos7/opensearch : Verify the roles of opensearch cluster nodes] ********************************************************************************************** changed: [os1] TASK [centos7/opensearch : Show the roles of opensearch cluster nodes] ************************************************************************************************ ok: [os1] => { "msg": "ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name\n10.0.10.96 22 96 14 0.23 0.15 0.06 di - os5\n10.0.10.254 17 95 14 0.28 0.19 0.08 di - os4\n10.0.10.243 20 95 25 0.40 0.25 0.10 dm - os2\n10.0.10.153 14 93 25 0.68 0.32 0.13 dm - os1\n10.0.10.79 6 96 15 0.23 0.16 0.07 dm * os3" } PLAY [opensearch dashboards installation & configuration] ************************************************************************************************************* TASK [Gathering Facts] ************************************************************************************************************************************************ ok: [dashboards1] TASK [centos7/dashboards : hostname] ********************************************************************************************************************************** changed: [dashboards1] TASK [centos7/dashboards : Install the libselinux-python for Amazon Linux 2] ****************************************************************************************** ok: [dashboards1] TASK [centos7/dashboards : Disable the selinux] *********************************************************************************************************************** ok: [dashboards1] TASK [centos7/dashboards : Hosts | populate inventory into hosts file] ************************************************************************************************ changed: [dashboards1] TASK [centos7/dashboards : Set open files limit in sysctl.conf] ******************************************************************************************************* ok: [dashboards1] TASK [centos7/dashboards : Dashboards Install | Download opensearch dashbaord 1.2.0] ********************************************************************************** changed: [dashboards1] TASK [centos7/dashboards : Dashboards Install | Create opensearch user] *********************************************************************************************** ok: [dashboards1] TASK [centos7/dashboards : Dashboards Install | Create home directory] ************************************************************************************************ changed: [dashboards1] TASK [centos7/dashboards : Dashboards Install | Extract the tar file] ************************************************************************************************* changed: [dashboards1] TASK [centos7/dashboards : Dashboards Install | Copy Configuration File] ********************************************************************************************** changed: [dashboards1] TASK [centos7/dashboards : Dashboards Install | create systemd service] *********************************************************************************************** changed: [dashboards1] TASK [centos7/dashboards : Make sure opensearch dashboards is started] ************************************************************************************************ changed: [dashboards1] TASK [centos7/dashboards : Get all the installed dashboards plugins] ************************************************************************************************** [WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo changed: [dashboards1] TASK [centos7/dashboards : Show all the installed dashboards plugins] ************************************************************************************************* ok: [dashboards1] => { "msg": "alertingDashboards@1.2.0.0\nanomalyDetectionDashboards@1.2.0.0\nganttChartDashboards@1.2.0.0\nindexManagementDashboards@1.2.0.0\nobservabilityDashboards@1.2.0.0\nqueryWorkbenchDashboards@1.2.0.0\nreportsDashboards@1.2.0.0\nsecurityDashboards@1.2.0.0" } PLAY RECAP ************************************************************************************************************************************************************ dashboards1 : ok=15 changed=9 unreachable=0 failed=0 os1 : ok=42 changed=33 unreachable=0 failed=0 os2 : ok=28 changed=22 unreachable=0 failed=0 os3 : ok=28 changed=22 unreachable=0 failed=0 os4 : ok=28 changed=22 unreachable=0 failed=0 os5 : ok=28 changed=22 unreachable=0 failed=0
saravanan30erd commented 2 years ago

@peterzhuamazon

  1. The folder name is still centos7, can we change to something else or remove the folder?

Changed to linux.

  1. Can you also update the README to include AL2? As of now only centos7 is mentioned.

Done

  1. I recently noticed that a lot of people still using Oracle Linux which is also RedHat based, if that can be test and support at some time it would be great.

Sure, created an issue #22

peterzhuamazon commented 2 years ago

Seems like there are some conflicts after my merge of #19. I will test this once the conflicts are resolved.

Thanks.

saravanan30erd commented 2 years ago

@peterzhuamazon merge conflicts are resolved. Please continue with test.

peterzhuamazon commented 2 years ago

@saravanan30erd we just released 1.2.2 and lets hold this for a bit so we can safely cut 1.2.2 tag. Will merge this next week when I get into the office. Same for Ubuntu.

Thanks.

saravanan30erd commented 2 years ago

@peterzhuamazon We need to merge this PR before any other PR's. Because this one have major change (folder changed from centos7 into linux) so it keep creating merge conflicts whenever new PR is merged and thus creating duplication work.

Could you please prioritize this one? May be we can have a call today to close this and ubuntu PR.

peterzhuamazon commented 2 years ago

Hi @saravanan30erd I am releasing 1.2.3 now so very little time on my side. Can we have a discussion for this later? Meanwhile, @TheAlgo if you can take a look it would be great.

Thanks.

peterzhuamazon commented 2 years ago

Update: I will update this PR today with more comments of testing. Able to find a small time window to do some tests. Sorry about delays.

peterzhuamazon commented 2 years ago 
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ModuleNotFoundError: No module named 'selinux'
fatal: [os2]: FAILED! => {"changed": false, "msg": "Failed to import the required Python library (libselinux-python) on os2's Python /usr/bin/python3.7. Please read the module documentation and install it in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter"}
saravanan30erd commented 2 years ago

@peterzhuamazon but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter may be its using wrong interpreter python 3 because in my case, its used python2.7 and I didn't face this issue. There was lot of issues due to different python versions since centos7/RHEL7 uses python2.7 only and AL2 have both python2.7 & 3, ubuntu only having python 3. so we need to stick with some stable ansible version which supports all. Can we have a call to check this?

peterzhuamazon commented 2 years ago

@peterzhuamazon but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter may be its using wrong interpreter python 3 because in my case, its used python2.7 and I didn't face this issue. There was lot of issues due to different python versions since centos7/RHEL7 uses python2.7 only and AL2 have both python2.7 & 3, ubuntu only having python 3. so we need to stick with some stable ansible version which supports all. Can we have a call to check this?

Reply email.

peterzhuamazon commented 2 years ago

Verified AL2:

IP  28 27 0 0.00 0.06 0.03 di - os5
IP  8 36 2 0.01 0.12 0.07 dm * os1
IP 26 27 0 0.00 0.08 0.05 di - os4
IP 27 27 0 0.00 0.06 0.03 dm - os2
IP 25 27 0 0.02 0.07 0.03 dm - os3

$ curl https://IP:9200/_cluster/health?pretty -u admin:admin --insecure
{
  "cluster_name" : "development-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 5,
  "number_of_data_nodes" : 5,
  "discovered_master" : true,
  "active_primary_shards" : 3,
  "active_shards" : 9,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}