AWX cannot set two vault credentials to the same job_template

[bherrin3]

ISSUE TYPE

• Bug Report

SUMMARY

• A git repo (SCM) has a sub-component repo that each have their own group_vars/all/credentials.yml vaults with different vault_secrets
• When attempting to add a second vault credential to a job_template via awx against Tower 3.6.4, the update fails with

  Response: {\"error\":\"Cannot assign multiple Vault credentials.\"}"}

• This behavior is seen in the GUI where the user cannot add two vault credentials to the same job_template in Tower 3.6.4
• The documentation in Tower maintains this should be possible: https://docs.ansible.com/ansible-tower/3.6.4/html/administration/multi-creds-assignment.html#ag-multi-vault

ENVIRONMENT

• AWX version: latest
• Ansible / Tower version: 2.9.1 / Tower 3.6.4
• Operating System: RHEL based
• Web Browser: N/A

STEPS TO REPRODUCE

AWX:

1. Latest Dynamic AT is deployed via IaC (Infrastructure as Code)
2. Attempt to assign two credentials to a job template - satlab-ansible-vault and satlab-admin-vault
3. The job_template assignment fails adding the job_template configuration

Tower:

1. Latest Dynamic AT is deployed via IaC
2. Attempt to assign two credentials to a job template - satlab-ansible-vault and satlab-admin-vault
3. Ansible Tower GUI disallows even selecting a second vault credential

EXPECTED RESULTS

• According to the documentation, this should be possible

ACTUAL RESULTS

 failed: [test-tower-end-to-end-bherring-MR80] (item={'name': 'satlab-admin-deploy-sat-lite-from-rhvm-02', 'description': 'Job Template used to deploy a base 100G, 6CPU, 20G Ram sat-lite template', 'project': 'satlab-admin-master', 'job_type': 'run', 'playbook': 'playbooks/utils/deploy-template-sat-lite.yml', 'ask_extra_vars': True, 'extra_vars': {'vm_root_password': '{{ default_satqe_password }}'}, 'concurrent_jobs_enabled': True, 'credentials': ['satlab-admin-vault', 'satlab-ansible-vault', 'machine-creds-with-jenkins-pvt-key', 'admin@internal-RHVM-02'], 'verbosity': 0, 'inventory': 'satlab-admin-inventory', 'state': 'present', 'custom_virtualenv': '/opt/virtualenvs/satlab-venv/'}) => {"ansible_loop_var": "job", "changed": false, "job": {"ask_extra_vars": true, "concurrent_jobs_enabled": true, "credentials": ["satlab-admin-vault", "satlab-ansible-vault", "machine-creds-with-jenkins-pvt-key", "admin@internal-RHVM-02"], "custom_virtualenv": "/opt/virtualenvs/satlab-venv/", "description": "Job Template used to deploy a base 100G, 6CPU, 20G Ram sat-lite template", "extra_vars": {"vm_root_password": "{{ default_satqe_password }}"}, "inventory": "satlab-admin-inventory", "job_type": "run", "name": "satlab-admin-deploy-sat-lite-from-rhvm-02", "playbook": "playbooks/utils/deploy-template-sat-lite.yml", "project": "satlab-admin-master", "state": "present", "verbosity": 0}, "msg": "Failed to add credential to job template: The Tower server claims it was sent a bad request.\n\nPOST https://dhcp-3-159.vms.sat.rdu2.redhat.com/api/v2/job_templates/18/credentials/\nParams: None\nData: {\"id\": 7, \"associate\": true}\n\nResponse: {\"error\":\"Cannot assign multiple Vault credentials.\"}"}
 [WARNING]: Failure using method (v2_runner_item_on_failed) in callback plugin
 (<ansible.plugins.callback.mail.CallbackModule object at 0x7f8126a1fb20>):
 'item'

ADDITIONAL INFORMATION

Please feel free to reach out or clarifiy any assumptions made.

This work is done via IaC, so we can work on scenarios or accept patches to see if solutions would work.

Available on IRC, gChat or anywhere required.

Thank you.

[AlanCoding]

Could you run this on higher verbosity?

I am extremely interested in this:

[WARNING]: Failure using method (v2_runner_item_on_failed) in callback plugin (<ansible.plugins.callback.mail.CallbackModule object at 0x7f8126a1fb20>): 'item'

That's probably a KeyError exception, and these are generally very high priority fixes when they come from the callback plugin. However, this doesn't give enough detail to know where things went wrong. If you run with -vvv verbosity or higher it should give a traceback, and I would really like to see that traceback.

[bherrin3]

--vvv gitlab-ci.yml job output

 failed: [test-tower-end-to-end-bherring-MR80] (item={'name': 'satlab-admin-deploy-sat-lite-from-rhvm-02', 'description': 'Job Template used to deploy a base 100G, 6CPU, 20G Ram sat-lite template', 'project': 'satlab-admin-master', 'job_type': 'run', 'playbook': 'playbooks/utils/deploy-template-sat-lite.yml', 'ask_extra_vars': True, 'extra_vars': {'vm_root_password': '{{ default_satqe_password }}'}, 'concurrent_jobs_enabled': True, 'credentials': ['satlab-admin-vault', 'satlab-ansible-vault', 'machine-creds-with-jenkins-pvt-key', 'admin@internal-RHVM-02'], 'verbosity': 0, 'inventory': 'satlab-admin-inventory', 'state': 'present', 'custom_virtualenv': '/opt/virtualenvs/satlab-venv/'}) => {
     "ansible_loop_var": "job",
     "changed": false,
     "invocation": {
         "module_args": {
             "ask_credential": false,
             "ask_diff_mode": false,
             "ask_extra_vars": true,
             "ask_inventory": false,
             "ask_job_type": false,
             "ask_limit": false,
             "ask_skip_tags": false,
             "ask_tags": false,
             "ask_verbosity": false,
             "become_enabled": false,
             "concurrent_jobs_enabled": true,
             "credential": "",
             "credentials": [
                 "satlab-admin-vault",
                 "satlab-ansible-vault",
                 "machine-creds-with-jenkins-pvt-key",
                 "admin@internal-RHVM-02"
             ],
             "custom_virtualenv": null,
             "description": "Job Template used to deploy a base 100G, 6CPU, 20G Ram sat-lite template",
             "diff_mode_enabled": false,
             "extra_vars": {
                 "vm_root_password": "{{ default_satqe_password }}"
             },
             "extra_vars_path": null,
             "fact_caching_enabled": false,
             "force_handlers_enabled": false,
             "forks": null,
             "host_config_key": "",
             "inventory": "satlab-admin-inventory",
             "job_tags": "",
             "job_type": "run",
             "limit": "",
             "name": "satlab-admin-deploy-sat-lite-from-rhvm-02",
             "playbook": "playbooks/utils/deploy-template-sat-lite.yml",
             "project": "satlab-admin-master",
             "skip_tags": "",
             "start_at_task": "",
             "survey_enabled": false,
             "survey_spec": null,
             "timeout": 0,
             "vault_credential": "",
             "verbosity": 0
         }
     },
     "job": {
         "ask_extra_vars": true,
         "concurrent_jobs_enabled": true,
         "credentials": [
             "satlab-admin-vault",
             "satlab-ansible-vault",
             "machine-creds-with-jenkins-pvt-key",
             "admin@internal-RHVM-02"
         ],
         "custom_virtualenv": "/opt/virtualenvs/satlab-venv/",
         "description": "Job Template used to deploy a base 100G, 6CPU, 20G Ram sat-lite template",
         "extra_vars": {
             "vm_root_password": "{{ default_satqe_password }}"
         },
         "inventory": "satlab-admin-inventory",
         "job_type": "run",
         "name": "satlab-admin-deploy-sat-lite-from-rhvm-02",
         "playbook": "playbooks/utils/deploy-template-sat-lite.yml",
         "project": "satlab-admin-master",
         "state": "present",
         "verbosity": 0
     },
     "msg": "Failed to add credential to job template: The Tower server claims it was sent a bad request.\n\nPOST https://dhcp-3-63.vms.sat.rdu2.redhat.com/api/v2/job_templates/18/credentials/\nParams: None\nData: {\"id\": 7, \"associate\": true}\n\nResponse: {\"error\":\"Cannot assign multiple Vault credentials.\"}"
 }
 [WARNING]: Failure using method (v2_runner_item_on_failed) in callback plugin
 (<ansible.plugins.callback.mail.CallbackModule object at 0x7f5df1ffae80>):
 'item'
 Callback Exception: 
   File "/usr/local/lib/python3.8/site-packages/ansible/executor/task_queue_manager.py", line 324, in send_callback
     method(*new_args, **kwargs)
    File "/usr/local/lib/python3.8/site-packages/ansible/plugins/callback/mail.py", line 234, in v2_runner_item_on_failed
     self.itembody += self.body_blob(json.dumps(result._result, cls=AnsibleJSONEncoder, indent=4), "failed item dump '%(item)s'" % result._result)
 Using module file /builds/bherring/satlab-admin/collections/ansible_collections/awx/awx/plugins/modules/tower_job_template.py
 Pipelining is enabled.
 <10.1.3.63> ESTABLISH SSH CONNECTION FOR USER: root
 <10.1.3.63> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="root"' -o ConnectTimeout=10 -o StrictHostKeyChecking=no -o ControlPath=/root/.ansible/cp/b841d14fc0 10.1.3.63 '/bin/sh -c '"'"'/usr/bin/python && sleep 0'"'"''
 <10.1.3.63> (1, b'\n{"msg": "Failed to add credential to job template: The Tower server claims it was sent a bad request.\\n\\nPOST https://dhcp-3-63.vms.sat.rdu2.redhat.com/api/v2/job_templates/19/credentials/\\nParams: None\\nData: {\\"id\\": 7, \\"associate\\": true}\\n\\nResponse: {\\"error\\":\\"Cannot assign multiple Vault credentials.\\"}", "failed": true, "exception": "WARNING: The below traceback may *not* be related to the actual failure.\\n  File \\"/tmp/ansible_awx.awx.tower_job_template_payload_z_ydP_/ansible_awx.awx.tower_job_template_payload.zip/ansible_collections/awx/awx/plugins/modules/tower_job_template.py\\", line 410, in main\\n  File \\"/usr/lib/python2.7/site-packages/tower_cli/resources/job_template.py\\", line 167, in associate_credential\\n    return self._assoc(\'credentials\', job_template, credential)\\n  File \\"/usr/lib/python2.7/site-packages/tower_cli/models/base.py\\", line 584, in _assoc\\n    r = client.post(url, data={\'associate\': True, \'id\': other})\\n  File \\"/usr/lib/python2.7/site-packages/requests/sessions.py\\", line 578, in post\\n    return self.request(\'POST\', url, data=data, json=json, **kwargs)\\n  File \\"/usr/lib/python2.7/site-packages/tower_cli/api.py\\", line 299, in request\\n    kwargs.get(\'data\', None), r.content.decode(\'utf8\'))\\n", "changed": false, "invocation": {"module_args": {"ask_diff_mode": false, "ask_limit": false, "custom_virtualenv": null, "job_type": "run", "ask_job_type": false, "skip_tags": "", "playbook": "playbooks/utils/deploy-template-sat-full.yml", "survey_enabled": false, "job_tags": "", "force_handlers_enabled": false, "inventory": "satlab-admin-inventory", "limit": "", "forks": null, "concurrent_jobs_enabled": true, "vault_credential": "", "diff_mode_enabled": false, "become_enabled": false, "credential": "", "ask_verbosity": false, "description": "Job Template used to deploy a base 100G, 6CPU, 20G Ram sat-full template", "fact_caching_enabled": false, "ask_skip_tags": false, "start_at_task": "", "ask_inventory": false, "host_config_key": "", "credentials": ["satlab-admin-vault", "satlab-ansible-vault", "machine-creds-with-jenkins-pvt-key", "admin@internal-RHVM-02"], "survey_spec": null, "ask_credential": false, "name": "satlab-admin-deploy-sat-full-from-rhvm-02", "extra_vars": {"vm_root_password": "{{ default_satqe_password }}"}, "verbosity": 0, "project": "satlab-admin-master", "ask_extra_vars": true, "extra_vars_path": null, "timeout": 0, "ask_tags": false}}}\n', b'')
 <10.1.3.63> Failed to connect to the host via ssh: 
 The full traceback is:
 WARNING: The below traceback may *not* be related to the actual failure.
   File "/tmp/ansible_awx.awx.tower_job_template_payload_z_ydP_/ansible_awx.awx.tower_job_template_payload.zip/ansible_collections/awx/awx/plugins/modules/tower_job_template.py", line 410, in main
   File "/usr/lib/python2.7/site-packages/tower_cli/resources/job_template.py", line 167, in associate_credential
     return self._assoc('credentials', job_template, credential)
   File "/usr/lib/python2.7/site-packages/tower_cli/models/base.py", line 584, in _assoc
     r = client.post(url, data={'associate': True, 'id': other})
   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 578, in post
     return self.request('POST', url, data=data, json=json, **kwargs)
   File "/usr/lib/python2.7/site-packages/tower_cli/api.py", line 299, in request
     kwargs.get('data', None), r.content.decode('utf8'))

Source being fed into awx.awx.tower_job_template:

  - name: satlab-admin-deploy-sat-lite-from-rhvm-02
    description: Job Template used to deploy a base 100G, 6CPU, 20G Ram sat-lite template
    project: satlab-admin-master
    job_type: run
    playbook: playbooks/utils/deploy-template-sat-lite.yml
    ask_extra_vars: True
    extra_vars:  "{{ lookup('file', './tower-configs/extra-vars-files/satlab-admin-deploy-baserhel-vars.json') }}"
    concurrent_jobs_enabled: true
    credentials:
     - satlab-admin-vault
     - satlab-ansible-vault
     - machine-creds-with-jenkins-pvt-key
     - admin@internal-RHVM-02
    verbosity: 0
    inventory: satlab-admin-inventory
    state: present
    custom_virtualenv: /opt/virtualenvs/satlab-venv/

[AlanCoding]

The bit about the callback plugin failure isn't associated with AWX or runner, but a bug in the mail callback plugin, and is probably generic to all loops with that. You could file that in the community.general collection issues.

Otherwise this error is looking pretty correct. It's saying that you have more than 1 vault credential without a vault_id in the list. Multiple vault credentials need ids for all of them.

[bherrin3]

@AlanCoding ack.

Note that the sub-component repo and the calling parent repo will need to use vault ids for this to work appropriately.

Will follow the documentation to re-key.

With that updated and verified, I will re-comment and close if that solves the issue.

Thanks!

Reference

• https://docs.ansible.com/ansible/latest/user_guide/vault.html?extIdCarryOver=true&sc_cid=701f2000001OH7EAAW#vault-ids-and-multiple-vault-passwords
• https://docs.ansible.com/ansible-tower/latest/html/administration/multi-creds-assignment.html

[bherrin3]

Update

Thanks for the information, @AlanCoding .

Finally finished the project move / refactor and I was able to add two vault-ids to the same tower-job_template.yml job to enable to secrets.

This worked once I got an IaC Ansible Tower deployed. However, for initial deploy using gitlab-ci.yml, I had to solve the chicken or the egg problem with credentials by not including the vault-ids within the ansible.cfg.

If you are IaC deploying with multiple vault-ids required, this is the only path.

Else after the successful AT deploy with ansible.cfg defined vault files, the SCM sync will fail as it cannot decrypt the vault to load SCM inventory with your other inventory (RHVM + SCM inventory in this case WITHOUT vault secret files in the repo).

The single path is:

1. Provide the vaults on the command line for all initial deployment. DO NOT include vault-ids in the ansible.cfg
2. Update all IaC in ATCasC to include the tower-credentials.yml definition of each credential with vault-id and then update all tower-job-templates.ymlto provide the correct vault credentials to decrypt the appropriate encrypt_string scoped variables for the job_templates / workflows.

Good stuff. Thanks for the hint!

Closing this out as resolved as two vault-id credentials can be added... but two non-vault ID vaults could not be added... even though the UI make it seem like that is possible... if you aren't familiar with vault-ids and their purpose. :)