search

HCL-TECH-SOFTWARE / connections-automation

Deployment and upgrade automation scripts for HCL Connections 7.0 based on Ansible
Apache License 2.0
17 stars 31 forks source link

Opensearch failed when repo list contains two opensearch entries #322

Open marde16 opened 8 months ago

marde16 commented 8 months ago

The playbook setup-component-pack-complete-harbor.yml failed during the opensearch deployment with the following output:

TASK [component-pack-harbor : Found opensearch chart version] ***************************************************************************************************************
ok: [k8.server.local] => {
    "msg": "2.7.0-20230921-162215\n2.7.0-20231006-050009"
}

TASK [component-pack-harbor : Record opensearch chart version] **************************************************************************************************************
skipping: [k8.server.local]

TASK [component-pack-harbor : Upgrade opensearch master] ********************************************************************************************************************
fatal: [k8.server.local]: FAILED! => {"changed": true, "cmd": ["helm", "upgrade", "opensearch-master", "v-connections-helm/opensearch", "-i", "--version", "2.7.0-20230921-162215", "2.7.0-20231006-050009", "-f", "/home/myuser/generated_charts/opensearch_master.yml", "--namespace", "connections", "--set", "common.probe.readinessProbe.timeoutSeconds=60", "--wait", "--timeout", "10m"], "delta": "0:00:00.041915", "end": "2023-10-27 14:43:58.855036", "msg": "non-zero return code", "rc": 1, "start": "2023-10-27 14:43:58.813121", "stderr": "Error: \"helm upgrade\" requires 2 arguments\n\nUsage:  helm upgrade [RELEASE] [CHART] [flags]", "stderr_lines": ["Error: \"helm upgrade\" requires 2 arguments", "", "Usage:  helm upgrade [RELEASE] [CHART] [flags]"], "stdout": "", "stdout_lines": []}

In our Component pack deployment the heml_repo_version_list looks as follows:

/home/myuser/generated_charts/helm_repo_version_list
NAME:CHART
v-connections-helm/bootstrap:0.1.0-20230918-122633
v-connections-helm/connections-env:0.1.40-20230918-122724
v-connections-helm/connections-outlook-desktop:0.1.0-20230918-122643
v-connections-helm/connections-persistent-stora...:0.1.1-20230921-155536
v-connections-helm/huddo-boards-cp:1.1.0-20230124-212221
v-connections-helm/huddo-boards-cp-activity-mig...:1.1.0-20230124-212221
v-connections-helm/infrastructure:0.1.0-20231006-050011
v-connections-helm/k8s-psp:0.1.0-20210909-112534
v-connections-helm/kudos-boards-cp:3.1.1-20221010-011453
v-connections-helm/kudos-boards-cp-activity-mig...:3.1.0-20221010-011453
v-connections-helm/mw-proxy:0.1.0-20230918-122709
v-connections-helm/opensearch:2.7.0-20230921-162215
v-connections-helm/opensearchstack:2.7.0-20231006-050009
v-connections-helm/orientme:0.1.0-20231006-050009
v-connections-helm/tailored-exp:1.0.0-20231006-050009
v-connections-helm/teams:1.0.0-20231006-050009

The following shell script returns both values for opensearch and opensearchstack "2.7.0-20230921-162215\n2.7.0-20231006-050009", why the playbook stops.

https://github.com/HCL-TECH-SOFTWARE/connections-automation/blob/f23269a34de6c3233941cc1d76b3e395fcfeb899/roles/hcl/component-pack-harbor/tasks/setup_opensearch.yml#L50-L53

In my opinion the shell script should be modified as follows:

shell: "cat {{ __ansible_cache_charts }}/helm_repo_version_list | grep "opensearch:" | grep -vE \"opensearch-\" | grep -vE opensearchstack | cut -d ':' -f2"

sabrina-yee commented 8 months ago

Hi @marde16, adding the : would work but just wondering if there is more going on here. The existing command seems to work using the sample helm_repo_version_list:

[myuser@mymachine ~]$ cat test.txt
NAME:CHART
v-connections-helm/bootstrap:0.1.0-20230918-122633
v-connections-helm/connections-env:0.1.40-20230918-122724
v-connections-helm/connections-outlook-desktop:0.1.0-20230918-122643
v-connections-helm/connections-persistent-stora...:0.1.1-20230921-155536
v-connections-helm/huddo-boards-cp:1.1.0-20230124-212221
v-connections-helm/huddo-boards-cp-activity-mig...:1.1.0-20230124-212221
v-connections-helm/infrastructure:0.1.0-20231006-050011
v-connections-helm/k8s-psp:0.1.0-20210909-112534
v-connections-helm/kudos-boards-cp:3.1.1-20221010-011453
v-connections-helm/kudos-boards-cp-activity-mig...:3.1.0-20221010-011453
v-connections-helm/mw-proxy:0.1.0-20230918-122709
v-connections-helm/opensearch:2.7.0-20230921-162215
v-connections-helm/opensearchstack:2.7.0-20231006-050009
v-connections-helm/orientme:0.1.0-20231006-050009
v-connections-helm/tailored-exp:1.0.0-20231006-050009
v-connections-helm/teams:1.0.0-20231006-050009
[myuser@mymachine ~]$
[myuser@mymachine ~]$ cat test.txt | grep opensearch | grep -vE "opensearch-" | grep -vE opensearchstack | cut -d ':' -f2
2.7.0-20230921-162215

By any chance there are more than 1 opensearch entries in your full helm_repo_version_list ?

stoeps13 commented 8 months ago

I checked my CR4 deployment:

[ansible@cnx7-ora-cp generated_charts]$ cat helm_repo_version_list
v-connections-helm/bootstrap:0.1.0-20230918-122633
v-connections-helm/connections-env:0.1.40-20230918-122724
v-connections-helm/connections-outlook-desktop:0.1.0-20230918-122643
v-connections-helm/connections-persistent-stora...:0.1.1-20230921-155536
v-connections-helm/huddo-boards-cp:1.1.0-20230124-212221
v-connections-helm/huddo-boards-cp-activity-mig...:1.1.0-20230124-212221
v-connections-helm/infrastructure:0.1.0-20231006-050011
v-connections-helm/k8s-psp:0.1.0-20210909-112534
v-connections-helm/kudos-boards-cp:3.1.1-20221010-011453
v-connections-helm/kudos-boards-cp-activity-mig...:3.1.0-20221010-011453
v-connections-helm/mw-proxy:0.1.0-20230918-122709
v-connections-helm/opensearch:2.7.0-20230921-162215
v-connections-helm/opensearchstack:2.7.0-20231006-050009
v-connections-helm/orientme:0.1.0-20231006-050009
v-connections-helm/tailored-exp:1.0.0-20231006-050009
v-connections-helm/teams:1.0.0-20231006-050009
[ansible@cnx7-ora-cp generated_charts]$ cat helm_repo_version_list | grep opensearch | grep -vE \"opensearch-\" | grep -vE opensearchstack | cut -d ':' -f2
2.7.0-20230921-162215

So it returns the opensearch version stamp.

grep -v removes matching strings, so when there is opensearch- or opensearchstack, these do not show up.

The : would be better, because the other two greps can be removed.:

cat helm_repo_version_list | grep opensearch: | cut -d ':' -f2
2.7.0-20230921-162215
sabrina-yee commented 8 months ago

Agreed, we can change it to : in the next push, but wanted to make sure there isn't another issue that caused the original grep to produce unexpected results. Thanks for confirming @stoeps13.

stoeps13 commented 8 months ago

One more thing @sabrina-yee , the helm_repo_version_list has 3 charts with abbreviations of ... in the end

v-connections-helm/connections-persistent-stora...:0.1.1-20230921-155536
v-connections-helm/huddo-boards-cp-activity-mig...:1.1.0-20230124-212221
v-connections-helm/kudos-boards-cp-activity-mig...:3.1.0-20221010-011453

So Huddo/Kudos will not use it, when Quay.io is configured, but I think the connections-persistent-storage could fail.

I'm wrong, it doesn't matter because the chart grep is for the shortened name: https://github.com/HCL-TECH-SOFTWARE/connections-automation/blob/f23269a34de6c3233941cc1d76b3e395fcfeb899/roles/hcl/component-pack-harbor/tasks/setup_connections_volumes.yml#L2

sabrina-yee commented 8 months ago

:) yes, we've seen it before so shorten the name. Hopefully these issue will all go away once we switch to using OCI (soon).

marde16 commented 8 months ago

Hi @sabrina-yee ,

I further analyzed my issue. The Ansible playbooks are based on 8CR2 version. Therefore the shell script differs between 8CR2 and the current version to get the opensearch version.

8CR2 version

- name:                      Get chart and version
  shell:                     "helm search repo {{ __helm_repository_local_name }} {{ __helm_repo_flag }} | grep opensearch | grep -vE \"opensearch-\" | awk {'print $2'}"
  register:                  opensearch_chart_version
  become_user:               "{{ __sudo_user }}"

since version 8CR3 version July 2023 cnx8 CR3 release https://github.com/HCL-TECH-SOFTWARE/connections-automation/blob/f23269a34de6c3233941cc1d76b3e395fcfeb899/roles/hcl/component-pack-harbor/tasks/setup_opensearch.yml#L50-L53

I have only two opensearch entries in my helm repo list:

v-connections-helm/opensearch:2.7.0-20230921-162215
v-connections-helm/opensearchstack:2.7.0-20231006-050009

So the grep parameters could be modified as @stoeps13 described.