Please confirm the following

[X] I agree to follow this project's code of conduct.
[X] I have checked the current issues for duplicates.
[X] I understand that the AWX Operator is open source software provided for free and that I might not receive a timely response.

Bug Summary

I have awx deployed on cloud instance running on k8s with persistent storage for postgresql database. As part of security practices, redeployed instance with same image and ipaddress. Able to use the existing persistent storage claim to deploy. Issue is when I try to launch job, getting error

2024-10-24 17:21:46,264 INFO [-] awx.main.wsrelay Starting producer for metrics 2024-10-24 17:22:29,849 WARNING [-] awx.main.tasks.system Heartbeat skew - interval=62.4549, expected=60 min_value in DecimalField should be Decimal type. 2024-10-24 17:22:30,020 INFO [-] awx.main.tasks.system Host awx-task-6f859bcf77-pfnhc Automatically Deprovisioned. 2024-10-24 17:31:24,789 INFO [8cbc4e8fcbff45aa9e1c50bc935d1754] awx.analytics.job_lifecycle job-9 waiting {"type": "job", "task_id": 9, "state": "waiting", "work_unit_id": nu ll, "task_name": "job_test_debug"} 2024-10-24 17:31:26,029 INFO [8cbc4e8fcbff45aa9e1c50bc935d1754] awx.analytics.job_lifecycle job-9 pre run {"type": "job", "task_id": 9, "state": "pre_run", "work_unit_id": nu ll, "task_name": "job_test_debug"} 2024-10-24 17:31:26,293 INFO [8cbc4e8fcbff45aa9e1c50bc935d1754] awx.analytics.job_lifecycle job-9 preparing playbook {"type": "job", "task_id": 9, "state": "preparing_playboo k", "work_unit_id": null, "task_name": "job_test_debug"} 2024-10-24 17:31:26,462 ERROR [8cbc4e8fcbff45aa9e1c50bc935d1754] awx.main.utils.encryption Failed to decrypt Credential(pk=3).password; if you've recently restored from a da tabase backup or are running in a clustered environment, check that your SECRET_KEY value is correct Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/cryptography/fernet.py", line 134, in _verify_signature h.verify(data[-32:]) cryptography.exceptions.InvalidSignature: Signature did not match digest.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/utils/encryption.py", line 159, in decrypt_field return smart_str(decrypt_value(key, value)) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/utils/encryption.py", line 136, in decrypt_value value = f.decrypt(encrypted) ^^^^^^^^^^^^^^^^^^^^ File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/cryptography/fernet.py", line 91, in decrypt return self._decrypt_data(data, timestamp, time_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/cryptography/fernet.py", line 152, in _decrypt_data self._verify_signature(data) File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/cryptography/fernet.py", line 136, in _verify_signature raise InvalidToken cryptography.fernet.InvalidToken 2024-10-24 17:31:26,516 ERROR [8cbc4e8fcbff45aa9e1c50bc935d1754] awx.main.tasks.jobs job 9 (running) Exception occurred while running task Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/cryptography/fernet.py", line 134, in _verify_signature h.verify(data[-32:]) cryptography.exceptions.InvalidSignature: Signature did not match digest.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/tasks/jobs.py", line 517, in run passwords = self.build_passwords(self.instance, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/tasks/jobs.py", line 855, in build_passwords value = runtime_passwords.get(field, cred.get_input('password' if field == 'ssh_password' else field, default='')) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/models/credential/init.py", line 283, in get_input return decrypt_field(self, field_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/utils/encryption.py", line 159, in decrypt_field return smart_str(decrypt_value(key, value))

File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/utils/encryption.py", line 136, in decrypt_value
value = f.decrypt(encrypted)
        ^^^^^^^^^^^^^^^^^^^^

File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/cryptography/fernet.py", line 91, in decrypt return self._decrypt_data(data, timestamp, time_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/cryptography/fernet.py", line 152, in _decrypt_data self._verify_signature(data) File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/cryptography/fernet.py", line 136, in _verify_signature raise InvalidToken cryptography.fernet.InvalidToken 2024-10-24 17:31:26,522 INFO [8cbc4e8fcbff45aa9e1c50bc935d1754] awx.analytics.job_lifecycle job-9 post run {"type": "job", "task_id": 9, "state": "post_run", "work_unit_id": null, "task_name": "job_test_debug"} 2024-10-24 17:31:26,895 INFO [8cbc4e8fcbff45aa9e1c50bc935d1754] awx.analytics.job_lifecycle job-9 finalize run {"type": "job", "task_id": 9, "state": "finalize_run", "work_unit_id": null, "task_name": "job_test_debug"}

My deployment details: OS: Debian12 kubernetes: kubectl version Client Version: v1.29.5 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.29.5

AWX operator deployment:

apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources:

releases

github.com/ansible/awx-operator/config/default?ref=2.18.0
- awx-task-pv.yaml

- awx-task-pvc.yaml

- awx-postgres-pv.yaml

- awx-postgres-pvc.yaml

- awx-deployment.yaml

Set the image tags to match the git version from above

images:

name: quay.io/ansible/awx-operator newTag: 2.18.0

Specify a custom namespace in which to install AWX

namespace: awx

AWX Operator version

2.18

AWX version

24.5.0

Kubernetes platform

kubernetes

Kubernetes/Platform version

kubectl version Client Version: v1.29.5 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.29.5

Modifications

yes

Steps to reproduce

apiVersion: awx.ansible.com/v1beta1 kind: AWX metadata: name: awx namespace: awx spec:

Awx secret for redploying

secret_key: "my-secret-key"

secret_key_secret: "IKM8WW2FxTfkAmck167vNJFzbOI76mQA" old_postgres_configuration_secret: "database-secret-key"

Custom nodeport 30786 for awx service

service_type: nodeport nodeport_port: 30786

Persistent volumes

postgres_storage_class: "manual" projects_persistence: true projects_existing_claim: "awx-web-pvc"

Expected results

Ansible job need to be executed without error

Actual results

File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/cryptography/fernet.py", line 152, in _decrypt_data self._verify_signature(data) File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/cryptography/fernet.py", line 136, in _verify_signature raise InvalidToken cryptography.fernet.InvalidToken

Additional information

No response

Operator Logs