opensearch-project / ansible-playbook

🤖 A community repository for Ansible Playbook of OpenSearch Project.
https://opensearch.org/
Apache License 2.0
86 stars 96 forks source link

[BUG][linux/opensearch : Security Plugin configuration | Initialize the opensearch security index in opensearch]task failed #34

Closed ripmars closed 2 years ago

ripmars commented 2 years ago

Describe the bug Failed with this playbook when run cluster install.All the opensearch nodes running.

TASK [linux/opensearch : Security Plugin configuration | Initialize the opensearch security index in opensearch]

bash /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh -cacert /usr/share/opensearch/config/root-ca.pem -cert /usr/share/opensearch/config/admin.pem -key /usr/share/opensearch/config/admin.key -f /usr/share/opensearch/plugins/opensearch-security/securityconfig/internal_users.yml -nhnv -icl -h 192.168.20.231

Security Admin v7
Will connect to 192.168.20.231:9300 ... done
Connected as CN=admin.gzhc.local,OU=Ops,O=gzhc.local\, Inc.,DC=gzhc.local
OpenSearch Version: 1.2.3
OpenSearch Security Version: 1.2.3.0
Diagnostic trace written to: /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin_diag_trace_2022-Jan-14_19-07-52.txt
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
ERR: Timed out while waiting for a green or yellow cluster state.
   * Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
   * Make also sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
   * If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
   * Add --accept-red-cluster to allow securityadmin to operate on a red cluster.

To Reproduce Steps to reproduce the behavior:

  1. inventories/opensearch/hosts file:
    
    gzites231 ansible_host=192.168.20.231  ansible_user=inception ip=192.168.20.231 roles=master,ingest
    gzites232 ansible_host=192.168.20.232  ansible_user=inception ip=192.168.20.232 roles=master,ingest
    gzites233 ansible_host=192.168.20.233  ansible_user=inception ip=192.168.20.233 roles=master,ingest
    gzites225 ansible_host=192.168.20.225  ansible_user=inception ip=192.168.20.225 roles=data
    gzites226 ansible_host=192.168.20.226  ansible_user=inception ip=192.168.20.226 roles=data
    gzites227 ansible_host=192.168.20.227  ansible_user=inception ip=192.168.20.227 roles=data
    gzites228 ansible_host=192.168.20.228  ansible_user=inception ip=192.168.20.228 roles=data
    gzites229 ansible_host=192.168.20.229  ansible_user=inception ip=192.168.20.229 roles=data
    gzites230 ansible_host=192.168.20.230  ansible_user=inception ip=192.168.20.230 roles=data

dashboards1 ansible_host=192.168.20.231 ansible_user=inception ip=192.168.20.231

List all the nodes in the os cluster

[os-cluster] gzites231 gzites232 gzites233 gzites225 gzites226 gzites227 gzites228 gzites229 gzites230

List all the Master eligible nodes under this group

[master] gzites231 gzites232 gzites233

[dashboards] dashboards1

2. The `inventories/opensearch/group_vars/all/all.yml` file:

Common opensearch configuration parameters

os_cluster_name: opensearch

opensearch download

os_download_url: http://192.168.20.23:8000/opensearch

opensearch version

os_version: "1.2.3"

opensearch dashboards version

os_dashboards_version: "1.2.0"

Configure hostnames for opensearch nodes

It is required to configure SSL

Example es1.example.com, es2.example.com

domain_name: gzhc.local

os_user: opensearch

Java memory heap values(GB) for opensearch

You can change it based on server specs

xms_value: 4 xmx_value: 4

Cluster type whether its single node or multi-node

cluster_type: multi-node

3. run the playbook `ansible-playbook -i inventories/opensearch/hosts opensearch.yml --extra-vars "admin_password=Pass@000 kibanaserver_password=Pass@000" -b --become-method=sudo -K -k` 
4. See error

**Expected behavior**
A clear and concise description of what you expected to happen.
The cluster should be initialized successfully,but not.
**Playbook Name**
Specify the Playbook which is affected?
TASK [linux/opensearch : Security Plugin configuration | Initialize the opensearch security index in opensearch] is affected.

**Host/Environment (please complete the following information):**
 - OS: Ubuntu 20.04
 - JDK: 11.0.12
 - Ansible Version: 2.11.7
 - Playbook Version: 1.1.0

**Additional context**

curl https://192.168.20.231:9200 -k

shown:

OpenSearch Security not initialized.

peterzhuamazon commented 2 years ago

@saravanan30erd any thoughts?

peterzhuamazon commented 2 years ago

@ripmars did you try to remove the related ansible/opensearch folder in /tmp and rerun it? I think this is caused due existing cache in /tmp, which cause the tarball not redownload. I think @saravanan30erd has a cleanup after the initial run complete, but not if you cancel in the middle of the run and rerun.

Thanks.

saravanan30erd commented 2 years ago

@ripmars As @peterzhuamazon said, previous steps wouldn't have properly run. Please provide the complete logs of whole playbook run.

ripmars commented 2 years ago

I run this before every reinstallation.

ansible es -i ites -u hcit -m shell -a 'systemctl stop opensearch;rm -f /tmp/opensearch.tar.gz; rm -rf /usr/share/opensearch  ; userdel -fr opensearch;rm -f /etc/systemd/system/opensearch.service ; systemctl daemon-reload' -k -K -b --become-method=sudo

and delete the ansible master host's '/tmp/opensearch-nodecert' folder.

ripmars commented 2 years ago

@saravanan30erd @peterzhuamazon Hey, my bad .my jvm settings are 4Gb,but I forget to upgrade the os ram to 8Gb. My lab is running now. Thanks for all.