ValueError: too many values to unpack (expected 2) on [installer : Check for presence of old awx Deployment]

e3b0c442 commented 11 months ago

Please confirm the following

[X] I agree to follow this project's code of conduct.
[X] I have checked the current issues for duplicates.
[X] I understand that the AWX Operator is open source software provided for free and that I might not receive a timely response.

Bug Summary

Installing an AWX cluster fails; logging from the operator indicates ValueError: too many values to unpack (expected 2) on step [installer : Check for presence of old awx Deployment]

AWX Operator version

2.9.0

AWX version

n/a (install failing)

Kubernetes platform

other (please specify in additional information)

Kubernetes/Platform version

k3s v1.27.8+k3s2

Modifications

no

Steps to reproduce

Install awx operator into a new, clean namespace
Install an awx cluster using the demo manifest in the docs
AWX fails to install

Expected results

AWX cluster is installed as expected

Actual results

No cluster, error in log

Additional information

---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx-demo
spec:
  service_type: nodeport

Operator Logs

--------------------------- Ansible Task StdOut -------------------------------

TASK [Verify imagePullSecrets] *************************************************
task path: /opt/ansible/playbooks/awx.yml:10

-------------------------------------------------------------------------------
{"level":"info","ts":"2023-12-17T02:45:34Z","logger":"proxy","msg":"Read object from cache","resource":{"IsResourceRequest":true,"Path":"/api/v1/namespaces/awx/secrets/redhat-operators-pull-secret","Verb":"get","APIPrefix":"api","APIGroup":"","APIVersion":"v1","Namespace":"awx","Resource":"secrets","Subresource":"","Name":"redhat-operators-pull-secret","Parts":["secrets","redhat-operators-pull-secret"]}}

--------------------------- Ansible Task StdOut -------------------------------

TASK [Create imagePullSecret] **************************************************
task path: /opt/ansible/playbooks/awx.yml:17

-------------------------------------------------------------------------------
{"level":"info","ts":"2023-12-17T02:45:34Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awx-demo","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWX","event_type":"playbook_on_task_start","job":"6748235340896356855","EventData.Name":"Create imagePullSecret"}

--------------------------- Ansible Task StdOut -------------------------------

{"level":"info","ts":"2023-12-17T02:45:34Z","logger":"logging_event_handler","msg":"[playbook debug]","name":"awx-demo","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWX","event_type":"runner_on_ok","job":"6748235340896356855","EventData.TaskArgs":""}
 TASK [debug] ******************************** 
ok: [localhost] => {
    "msg": "CLUSTER TYPE: is_openshift=False; is_k8s=True"
}

-------------------------------------------------------------------------------
{"level":"info","ts":"2023-12-17T02:45:34Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awx-demo","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWX","event_type":"playbook_on_task_start","job":"6748235340896356855","EventData.Name":"installer : Check for presence of old awx Deployment"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [installer : Check for presence of old awx Deployment] ********************
task path: /opt/ansible/roles/installer/tasks/main.yml:2

-------------------------------------------------------------------------------
{"level":"info","ts":"2023-12-17T02:45:35Z","logger":"proxy","msg":"Cache miss: apps/v1, Kind=Deployment, awx/awx-demo"}
{"level":"info","ts":"2023-12-17T02:45:35Z","logger":"logging_event_handler","msg":"[playbook task start]","name":"awx-demo","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWX","event_type":"playbook_on_task_start","job":"6748235340896356855","EventData.Name":"installer : Check for presence of awx-task Deployment"}

--------------------------- Ansible Task StdOut -------------------------------

TASK [installer : Check for presence of awx-task Deployment] *******************
task path: /opt/ansible/roles/installer/tasks/main.yml:10

-------------------------------------------------------------------------------
{"level":"error","ts":"2023-12-17T02:45:35Z","logger":"logging_event_handler","msg":"","name":"awx-demo","namespace":"awx","gvk":"awx.ansible.com/v1beta1, Kind=AWX","event_type":"runner_on_failed","job":"6748235340896356855","EventData.Task":"Check for presence of awx-task Deployment","EventData.TaskArgs":"","EventData.FailedTaskPath":"/opt/ansible/roles/installer/tasks/main.yml:10","error":"[playbook task failed]","stacktrace":"github.com/operator-framework/ansible-operator-plugins/internal/ansible/events.loggingEventHandler.Handle\n\tansible-operator-plugins/internal/ansible/events/log_events.go:111"}

--------------------------- Ansible Task StdOut -------------------------------

 TASK [Check for presence of awx-task Deployment] ******************************** 
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ValueError: too many values to unpack (expected 2)
fatal: [localhost]: FAILED! => {"changed": false, "module_stderr": "Traceback (most recent call last):\n  File \"/opt/ansible/.ansible/tmp/ansible-tmp-1702781135.2775533-2088-44727393974459/AnsiballZ_k8s_info.py\", line 107, in <module>\n    _ansiballz_main()\n  File \"/opt/ansible/.ansible/tmp/ansible-tmp-1702781135.2775533-2088-44727393974459/AnsiballZ_k8s_info.py\", line 99, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/opt/ansible/.ansible/tmp/ansible-tmp-1702781135.2775533-2088-44727393974459/AnsiballZ_k8s_info.py\", line 47, in invoke_module\n    runpy.run_module(mod_name='ansible_collections.kubernetes.core.plugins.modules.k8s_info', init_globals=dict(_module_fqn='ansible_collections.kubernetes.core.plugins.modules.k8s_info', _modlib_path=modlib_path),\n  File \"/usr/lib64/python3.9/runpy.py\", line 225, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib64/python3.9/runpy.py\", line 97, in _run_module_code\n    _run_code(code, mod_globals, init_globals,\n  File \"/usr/lib64/python3.9/runpy.py\", line 87, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_k8s_info_payload_6rbz_77r/ansible_k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/modules/k8s_info.py\", line 229, in <module>\n  File \"/tmp/ansible_k8s_info_payload_6rbz_77r/ansible_k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/modules/k8s_info.py\", line 223, in main\n  File \"/tmp/ansible_k8s_info_payload_6rbz_77r/ansible_k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/modules/k8s_info.py\", line 183, in execute_module\n  File \"/tmp/ansible_k8s_info_payload_6rbz_77r/ansible_k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/k8s/service.py\", line 248, in find\n  File \"/tmp/ansible_k8s_info_payload_6rbz_77r/ansible_k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/k8s/service.py\", line 85, in find_resource\n  File \"/tmp/ansible_k8s_info_payload_6rbz_77r/ansible_k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/k8s/client.py\", line 305, in resource\n  File \"/tmp/ansible_k8s_info_payload_6rbz_77r/ansible_k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/k8s/client.py\", line 285, in _find_resource_with_prefix\n  File \"/tmp/ansible_k8s_info_payload_6rbz_77r/ansible_k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/client/discovery.py\", line 158, in get\n  File \"/usr/local/lib/python3.9/site-packages/kubernetes/dynamic/discovery.py\", line 246, in search\n    results = self.__search(self.__build_search(**kwargs), self.__resources, [])\n  File \"/usr/local/lib/python3.9/site-packages/kubernetes/dynamic/discovery.py\", line 294, in __search\n    matches.extend(self.__search([key] + parts[1:], resources, reqParams))\n  File \"/usr/local/lib/python3.9/site-packages/kubernetes/dynamic/discovery.py\", line 280, in __search\n    return self.__search(parts[1:], resourcePart, reqParams + [part] )\n  File \"/usr/local/lib/python3.9/site-packages/kubernetes/dynamic/discovery.py\", line 294, in __search\n    matches.extend(self.__search([key] + parts[1:], resources, reqParams))\n  File \"/usr/local/lib/python3.9/site-packages/kubernetes/dynamic/discovery.py\", line 280, in __search\n    return self.__search(parts[1:], resourcePart, reqParams + [part] )\n  File \"/usr/local/lib/python3.9/site-packages/kubernetes/dynamic/discovery.py\", line 269, in __search\n    resourcePart.resources = self.get_resources_for_api_version(\n  File \"/tmp/ansible_k8s_info_payload_6rbz_77r/ansible_k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/client/discovery.py\", line 116, in get_resources_for_api_version\nValueError: too many values to unpack (expected 2)\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

-------------------------------------------------------------------------------
{"level":"error","ts":"2023-12-17T02:45:35Z","logger":"runner","msg":"ansible-playbook [core 2.15.3]\r\n  config file = /etc/ansible/ansible.cfg\r\n  configured module search path = ['/usr/share/ansible/openshift']\r\n  ansible python module location = /usr/local/lib/python3.9/site-packages/ansible\r\n  ansible collection location = /opt/ansible/.ansible/collections:/usr/share/ansible/collections\r\n  executable location = /usr/local/bin/ansible-playbook\r\n  python version = 3.9.16 (main, May 31 2023, 12:21:58) [GCC 8.5.0 20210514 (Red Hat 8.5.0-18)] (/usr/bin/python3)\r\n  jinja version = 3.1.2\r\n  libyaml = True\r\nUsing /etc/ansible/ansible.cfg as config file\r\nSkipping callback 'awx_display', as we already have a stdout callback.\nSkipping callback 'default', as we already have a stdout callback.\nSkipping callback 'minimal', as we already have a stdout callback.\nSkipping callback 'oneline', as we already have a stdout callback.\n\r\nPLAYBOOK: awx.yml **************************************************************\n1 plays in /opt/ansible/playbooks/awx.yml\n[WARNING]: Found variable using reserved name: no_log\n\r\nPLAY [localhost] ***************************************************************\n\r\nTASK [Verify imagePullSecrets] *************************************************\r\ntask path: /opt/ansible/playbooks/awx.yml:10\nok: [localhost] => {\"censored\": \"the output has been hidden due to the fact that 'no_log: true' was specified for this result\", \"changed\": false}\n\r\nTASK [Create imagePullSecret] **************************************************\r\ntask path: /opt/ansible/playbooks/awx.yml:17\nskipping: [localhost] => {\"changed\": false, \"false_condition\": \"(_rh_ops_secret is not defined) or not (_rh_ops_secret['resources'] | length)\", \"skip_reason\": \"Conditional result was False\"}\n\r\nTASK [common : Get information about the cluster] ******************************\r\ntask path: /opt/ansible/roles/common/tasks/main.yml:3\nredirecting (type: lookup) ansible.builtin.k8s to kubernetes.core.k8s\nok: [localhost] => {\"ansible_facts\": {\"api_groups\": [\"\", \"apiregistration.k8s.io\", \"apps\", \"events.k8s.io\", \"authentication.k8s.io\", \"authorization.k8s.io\", \"autoscaling\", \"batch\", \"certificates.k8s.io\", \"networking.k8s.io\", \"policy\", \"rbac.authorization.k8s.io\", \"storage.k8s.io\", \"admissionregistration.k8s.io\", \"apiextensions.k8s.io\", \"scheduling.k8s.io\", \"coordination.k8s.io\", \"node.k8s.io\", \"discovery.k8s.io\", \"flowcontrol.apiserver.k8s.io\", \"acme.cert-manager.io\", \"apm.k8s.elastic.co\", \"cert-manager.io\", \"databases.spotahome.com\", \"deviceplugin.intel.com\", \"elasticsearch.k8s.elastic.co\", \"enterprisesearch.k8s.elastic.co\", \"gateway.networking.k8s.io\", \"helm.cattle.io\", \"k3s.cattle.io\", \"kibana.k8s.elastic.co\", \"kubevirt.io\", \"kustomize.toolkit.fluxcd.io\", \"monitoring.coreos.com\", \"notification.toolkit.fluxcd.io\", \"onepassword.com\", \"security.istio.io\", \"snapshot.storage.k8s.io\", \"source.toolkit.fluxcd.io\", \"subresources.kubevirt.io\", \"upgrade.cattle.io\", \"agent.k8s.elastic.co\", \"argoproj.io\", \"autoscaling.k8s.elastic.co\", \"clone.kubevirt.io\", \"export.kubevirt.io\", \"extensions.istio.io\", \"external-secrets.io\", \"generators.external-secrets.io\", \"install.istio.io\", \"instancetype.kubevirt.io\", \"logstash.k8s.elastic.co\", \"maps.k8s.elastic.co\", \"migrations.kubevirt.io\", \"nfd.k8s-sigs.io\", \"pool.kubevirt.io\", \"snapshot.kubevirt.io\", \"stackconfigpolicy.k8s.elastic.co\", \"telemetry.istio.io\", \"traefik.containo.us\", \"traefik.io\", \"networking.istio.io\", \"awx.ansible.com\", \"beat.k8s.elastic.co\", \"image.toolkit.fluxcd.io\", \"longhorn.io\", \"postgres-operator.crunchydata.com\", \"redis.redis.opstreelabs.in\", \"fpga.intel.com\", \"pgv2.percona.com\", \"k8s.keycloak.org\", \"helm.toolkit.fluxcd.io\", \"metrics.k8s.io\"]}, \"changed\": false}\n\r\nTASK [common : Determine the cluster type] *************************************\r\ntask path: /opt/ansible/roles/common/tasks/main.yml:10\nok: [localhost] => {\"ansible_facts\": {\"is_k8s\": true, \"is_openshift\": false}, \"changed\": false}\n\r\nTASK [common : debug] **********************************************************\r\ntask path: /opt/ansible/roles/common/tasks/main.yml:19\nok: [localhost] => {\r\n    \"msg\": \"CLUSTER TYPE: is_openshift=False; is_k8s=True\"\r\n}\n\r\nTASK [installer : Check for presence of old awx Deployment] ********************\r\ntask path: /opt/ansible/roles/installer/tasks/main.yml:2\nok: [localhost] => {\"api_found\": true, \"changed\": false, \"resources\": []}\n\r\nTASK [installer : Check for presence of awx-task Deployment] *******************\r\ntask path: /opt/ansible/roles/installer/tasks/main.yml:10\nAn exception occurred during task execution. To see the full traceback, use -vvv. The error was: ValueError: too many values to unpack (expected 2)\r\nfatal: [localhost]: FAILED! => {\"changed\": false, \"module_stderr\": \"Traceback (most recent call last):\\n  File \\\"/opt/ansible/.ansible/tmp/ansible-tmp-1702781135.2775533-2088-44727393974459/AnsiballZ_k8s_info.py\\\", line 107, in <module>\\n    _ansiballz_main()\\n  File \\\"/opt/ansible/.ansible/tmp/ansible-tmp-1702781135.2775533-2088-44727393974459/AnsiballZ_k8s_info.py\\\", line 99, in _ansiballz_main\\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\\n  File \\\"/opt/ansible/.ansible/tmp/ansible-tmp-1702781135.2775533-2088-44727393974459/AnsiballZ_k8s_info.py\\\", line 47, in invoke_module\\n    runpy.run_module(mod_name='ansible_collections.kubernetes.core.plugins.modules.k8s_info', init_globals=dict(_module_fqn='ansible_collections.kubernetes.core.plugins.modules.k8s_info', _modlib_path=modlib_path),\\n  File \\\"/usr/lib64/python3.9/runpy.py\\\", line 225, in run_module\\n    return _run_module_code(code, init_globals, run_name, mod_spec)\\n  File \\\"/usr/lib64/python3.9/runpy.py\\\", line 97, in _run_module_code\\n    _run_code(code, mod_globals, init_globals,\\n  File \\\"/usr/lib64/python3.9/runpy.py\\\", line 87, in _run_code\\n    exec(code, run_globals)\\n  File \\\"/tmp/ansible_k8s_info_payload_6rbz_77r/ansible_k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/modules/k8s_info.py\\\", line 229, in <module>\\n  File \\\"/tmp/ansible_k8s_info_payload_6rbz_77r/ansible_k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/modules/k8s_info.py\\\", line 223, in main\\n  File \\\"/tmp/ansible_k8s_info_payload_6rbz_77r/ansible_k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/modules/k8s_info.py\\\", line 183, in execute_module\\n  File \\\"/tmp/ansible_k8s_info_payload_6rbz_77r/ansible_k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/k8s/service.py\\\", line 248, in find\\n  File \\\"/tmp/ansible_k8s_info_payload_6rbz_77r/ansible_k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/k8s/service.py\\\", line 85, in find_resource\\n  File \\\"/tmp/ansible_k8s_info_payload_6rbz_77r/ansible_k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/k8s/client.py\\\", line 305, in resource\\n  File \\\"/tmp/ansible_k8s_info_payload_6rbz_77r/ansible_k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/k8s/client.py\\\", line 285, in _find_resource_with_prefix\\n  File \\\"/tmp/ansible_k8s_info_payload_6rbz_77r/ansible_k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/client/discovery.py\\\", line 158, in get\\n  File \\\"/usr/local/lib/python3.9/site-packages/kubernetes/dynamic/discovery.py\\\", line 246, in search\\n    results = self.__search(self.__build_search(**kwargs), self.__resources, [])\\n  File \\\"/usr/local/lib/python3.9/site-packages/kubernetes/dynamic/discovery.py\\\", line 294, in __search\\n    matches.extend(self.__search([key] + parts[1:], resources, reqParams))\\n  File \\\"/usr/local/lib/python3.9/site-packages/kubernetes/dynamic/discovery.py\\\", line 280, in __search\\n    return self.__search(parts[1:], resourcePart, reqParams + [part] )\\n  File \\\"/usr/local/lib/python3.9/site-packages/kubernetes/dynamic/discovery.py\\\", line 294, in __search\\n    matches.extend(self.__search([key] + parts[1:], resources, reqParams))\\n  File \\\"/usr/local/lib/python3.9/site-packages/kubernetes/dynamic/discovery.py\\\", line 280, in __search\\n    return self.__search(parts[1:], resourcePart, reqParams + [part] )\\n  File \\\"/usr/local/lib/python3.9/site-packages/kubernetes/dynamic/discovery.py\\\", line 269, in __search\\n    resourcePart.resources = self.get_resources_for_api_version(\\n  File \\\"/tmp/ansible_k8s_info_payload_6rbz_77r/ansible_k8s_info_payload.zip/ansible_collections/kubernetes/core/plugins/module_utils/client/discovery.py\\\", line 116, in get_resources_for_api_version\\nValueError: too many values to unpack (expected 2)\\n\", \"module_stdout\": \"\", \"msg\": \"MODULE FAILURE\\nSee stdout/stderr for the exact error\", \"rc\": 1}\n\r\nPLAY RECAP *********************************************************************\r\nlocalhost                  : ok=5    changed=0    unreachable=0    failed=1    skipped=1    rescued=0    ignored=0   \n","job":"6748235340896356855","name":"awx-demo","namespace":"awx","error":"exit status 2","stacktrace":"github.com/operator-framework/ansible-operator-plugins/internal/ansible/runner.(*runner).Run.func1\n\tansible-operator-plugins/internal/ansible/runner/runner.go:269"}

----- Ansible Task Status Event StdOut (awx.ansible.com/v1beta1, Kind=AWX, awx-demo/awx) -----

PLAY RECAP *********************************************************************
localhost                  : ok=5    changed=0    unreachable=0    failed=1    skipped=1    rescued=0    ignored=0   

----------
{"level":"error","ts":"2023-12-17T02:45:35Z","msg":"Reconciler error","controller":"awx-controller","object":{"name":"awx-demo","namespace":"awx"},"namespace":"awx","name":"awx-demo","reconcileID":"f2e0b74d-e9c7-4bd1-93d1-262ec2457c2f","error":"event runner on failed","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:235"}

e3b0c442 commented 11 months ago

I was unable to duplicate on a clean k3s install on a new VM.

Further digging led me to this issue: https://github.com/ansible-collections/kubernetes.core/issues/659, which was interesting to me because the AWX operator had worked previously for me, but I recently added Kubevirt to my cluster.

From there I was able to determine that there are several places where the incorrect api_version v1 is used to refer to the Deployment resource, instead of the correct api_version apps/v1.

Once I corrected those, I was able to successfully deploy AWX with the operator.

I will work with my employer to get permission to open a contribution PR, but that may take time; if somebody else wants to address, the following lines are the offending usages of v1:

https://github.com/ansible/awx-operator/blob/582701d94971480978b568e5f281f0ef38ebc0e5/roles/installer/tasks/install.yml#L5 https://github.com/ansible/awx-operator/blob/582701d94971480978b568e5f281f0ef38ebc0e5/roles/installer/tasks/main.yml#L12 https://github.com/ansible/awx-operator/blob/582701d94971480978b568e5f281f0ef38ebc0e5/roles/installer/tasks/main.yml#L20

Hope this helps, and thanks for the efforts here.