Open shaikabdulm opened 1 day ago
Verified the secret key is same after re-deploying awx
kubectl -n awx exec -i -t awx-task-6f859bcf77-6bnv5 -c awx-task -- bash bash-5.1# awx-manage shell_plus --quiet Python 3.11.7 (main, Jan 22 2024, 00:00:00) [GCC 11.4.1 20231218 (Red Hat 11.4.1-3)] on linux Type "help", "copyright", "credits" or "license" for more information. (InteractiveConsole)
settings.SECRET_KEY b'IKM8WW2FxTfkAmck167vNJFzbOI76mQA' quit() bash-5.1# cat /etc/tower/SECRET_KEY IKM8WW2FxTfkAmck167vNJFzbOI76mQA bash-5.1#
Created secrets using same manifest files before re-deployment
cat awx-postgres-configuration.yaml apiVersion: v1 kind: Secret type: Opaque data:
password: "my-secret-database" database: "YXd4" host: "YXd4LXBvc3RncmVzLTE1" port: "NTQzMg==" type: "bWFuYWdlZA==" username: "YXd4" metadata: name: awx-postgres-configuration namespace: awx labels: app.kubernetes.io/component: awx app.kubernetes.io/managed-by: awx-operator app.kubernetes.io/operator-version: 2.18.0 app.kubernetes.io/part-of: awx name: awx-postgres-configuration namespace: awx
Please confirm the following
Bug Summary
I have awx deployed on cloud instance running on k8s with persistent storage for postgresql database. As part of security practices, redeployed instance with same image and ipaddress. Able to use the existing persistent storage claim to deploy. Issue is when I try to launch job, getting error
2024-10-24 17:21:46,264 INFO [-] awx.main.wsrelay Starting producer for metrics 2024-10-24 17:22:29,849 WARNING [-] awx.main.tasks.system Heartbeat skew - interval=62.4549, expected=60 min_value in DecimalField should be Decimal type. 2024-10-24 17:22:30,020 INFO [-] awx.main.tasks.system Host awx-task-6f859bcf77-pfnhc Automatically Deprovisioned. 2024-10-24 17:31:24,789 INFO [8cbc4e8fcbff45aa9e1c50bc935d1754] awx.analytics.job_lifecycle job-9 waiting {"type": "job", "task_id": 9, "state": "waiting", "work_unit_id": nu ll, "task_name": "job_test_debug"} 2024-10-24 17:31:26,029 INFO [8cbc4e8fcbff45aa9e1c50bc935d1754] awx.analytics.job_lifecycle job-9 pre run {"type": "job", "task_id": 9, "state": "pre_run", "work_unit_id": nu ll, "task_name": "job_test_debug"} 2024-10-24 17:31:26,293 INFO [8cbc4e8fcbff45aa9e1c50bc935d1754] awx.analytics.job_lifecycle job-9 preparing playbook {"type": "job", "task_id": 9, "state": "preparing_playboo k", "work_unit_id": null, "task_name": "job_test_debug"} 2024-10-24 17:31:26,462 ERROR [8cbc4e8fcbff45aa9e1c50bc935d1754] awx.main.utils.encryption Failed to decrypt
Credential(pk=3).password
; if you've recently restored from a da tabase backup or are running in a clustered environment, check that yourSECRET_KEY
value is correct Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/cryptography/fernet.py", line 134, in _verify_signature h.verify(data[-32:]) cryptography.exceptions.InvalidSignature: Signature did not match digest.During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/utils/encryption.py", line 159, in decrypt_field return smart_str(decrypt_value(key, value)) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/utils/encryption.py", line 136, in decrypt_value value = f.decrypt(encrypted) ^^^^^^^^^^^^^^^^^^^^ File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/cryptography/fernet.py", line 91, in decrypt return self._decrypt_data(data, timestamp, time_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/cryptography/fernet.py", line 152, in _decrypt_data self._verify_signature(data) File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/cryptography/fernet.py", line 136, in _verify_signature raise InvalidToken cryptography.fernet.InvalidToken 2024-10-24 17:31:26,516 ERROR [8cbc4e8fcbff45aa9e1c50bc935d1754] awx.main.tasks.jobs job 9 (running) Exception occurred while running task Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/cryptography/fernet.py", line 134, in _verify_signature h.verify(data[-32:]) cryptography.exceptions.InvalidSignature: Signature did not match digest.
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/tasks/jobs.py", line 517, in run passwords = self.build_passwords(self.instance, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/tasks/jobs.py", line 855, in build_passwords value = runtime_passwords.get(field, cred.get_input('password' if field == 'ssh_password' else field, default='')) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/models/credential/init.py", line 283, in get_input return decrypt_field(self, field_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/utils/encryption.py", line 159, in decrypt_field return smart_str(decrypt_value(key, value))
File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/cryptography/fernet.py", line 91, in decrypt return self._decrypt_data(data, timestamp, time_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/cryptography/fernet.py", line 152, in _decrypt_data self._verify_signature(data) File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/cryptography/fernet.py", line 136, in _verify_signature raise InvalidToken cryptography.fernet.InvalidToken 2024-10-24 17:31:26,522 INFO [8cbc4e8fcbff45aa9e1c50bc935d1754] awx.analytics.job_lifecycle job-9 post run {"type": "job", "task_id": 9, "state": "post_run", "work_unit_id": null, "task_name": "job_test_debug"} 2024-10-24 17:31:26,895 INFO [8cbc4e8fcbff45aa9e1c50bc935d1754] awx.analytics.job_lifecycle job-9 finalize run {"type": "job", "task_id": 9, "state": "finalize_run", "work_unit_id": null, "task_name": "job_test_debug"}
My deployment details: OS: Debian12 kubernetes: kubectl version Client Version: v1.29.5 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.29.5
AWX operator deployment:
apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources:
Find the latest tag here: https://github.com/ansible/awx-operator/releases
- awx-task-pv.yaml
- awx-task-pvc.yaml
- awx-postgres-pv.yaml
- awx-postgres-pvc.yaml
- awx-deployment.yaml
Set the image tags to match the git version from above
images:
Specify a custom namespace in which to install AWX
namespace: awx
AWX Operator version
2.18
AWX version
24.5.0
Kubernetes platform
kubernetes
Kubernetes/Platform version
kubectl version Client Version: v1.29.5 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.29.5
Modifications
yes
Steps to reproduce
apiVersion: awx.ansible.com/v1beta1 kind: AWX metadata: name: awx namespace: awx spec:
Awx secret for redploying
secret_key: "my-secret-key"
secret_key_secret: "IKM8WW2FxTfkAmck167vNJFzbOI76mQA" old_postgres_configuration_secret: "database-secret-key"
Custom nodeport 30786 for awx service
service_type: nodeport nodeport_port: 30786
Persistent volumes
postgres_storage_class: "manual" projects_persistence: true projects_existing_claim: "awx-web-pvc"
Expected results
Ansible job need to be executed without error
Actual results
File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/cryptography/fernet.py", line 152, in _decrypt_data self._verify_signature(data) File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/cryptography/fernet.py", line 136, in _verify_signature raise InvalidToken cryptography.fernet.InvalidToken
Additional information
No response
Operator Logs
No response