ansible-collections / cisco.iosxr

Ansible Network Collection for Cisco IOSXR
GNU General Public License v3.0
67 stars 47 forks source link

iosxr_command module should support answering a repeated prompt #317

Closed ialonsob closed 1 year ago

ialonsob commented 1 year ago
SUMMARY

The iosxr_command module should support answering to repeated prompts in a command.

ISSUE TYPE
COMPONENT NAME

iosxr_command ansible module.

ADDITIONAL INFORMATION

Certain exec commands require answering twice a prompt to confirm user input.

Example command with a confirm prompt:

RP/0/RP0/CPU0:XR9KV#crypto ca enroll MSCA
Thu Dec  8 12:47:47.414 UTC
% Start certificate enrollment ... 
% Create a challenge password. You will need to verbally provide this
  password to the CA Administrator in order to revoke your certificate.
% For security reasons your password will not be saved in the configuration.
% Please make a note of it.

Password: 
Re-enter Password: 

% The subject name in the certificate will include: CN=XRV01.cxlab.cisco.com
(..)

The following ansible play trying to execute the command with two prompts is timing out:

- name: Enroll the IOS-XR Routers with the Intermediate CA
  hosts: ROUTERS
  gather_facts: False
  connection: local
  tasks:
    - name: Enroll with the Intermediate CA via SCEP
      cisco.iosxr.iosxr_command:
        commands:
          - {command: crypto ca enroll MSCA, prompt: "Password:", answer: "password\r"}

The following ansible play sends two carrier returns, i.e., setting the password to none in the CSR request. This play works fine:

- name: Enroll the IOS-XR Routers with the Intermediate CA
  hosts: ROUTERS
  gather_facts: False
  connection: local
  tasks:
    - name: Enroll with the Intermediate CA via SCEP
      cisco.iosxr.iosxr_command:
        commands:
          - {command: crypto ca enroll MSCA, prompt: "Password:", answer: "\r\r"}
ashwini-mhatre commented 1 year ago

@ialonsob can you try like following example

- name: Reload
  cisco.iosxr.iosxr_command:
      commands:
          - {command: reload warm, prompt: ["Proceed? [confirm]", "Proceed with reload?  [confirm]" ], answer: ["\\r", "\\r"] }
ialonsob commented 1 year ago

Thanks @ashwini-mhatre

I've tried a couple of options using lists of prompts and answers, here below the results that I'm getting

Setting the answers to "\r" The task times out.

    - name: Enroll with the Intermediate CA via SCEP
      cisco.iosxr.iosxr_command:
        commands:
          - {command: crypto ca enroll MSCA, prompt: ["Password:", "Re-enter Password:"], answer: ["\r", "\r"] }

TASK [Enroll with the Intermediate CA via SCEP] ****************************************************************************************************************************************************************************************
fatal: [XR9KV02]: FAILED! => {"changed": false, "msg": "command timeout triggered, timeout value is 30 secs.\nSee the timeout setting options in the Network Debug and Troubleshooting Guide."}
fatal: [XR9KV01]: FAILED! => {"changed": false, "msg": "command timeout triggered, timeout value is 30 secs.\nSee the timeout setting options in the Network Debug and Troubleshooting Guide."}

Setting the answers to "password\r" After multiple runs, this option has worked only once. When it fails, in the device PKI debug, the error raised is the same than if the two passwords provided are different, see example below.

    - name: Enroll with the Intermediate CA via SCEP
      cisco.iosxr.iosxr_command:
        commands:
          - {command: crypto ca enroll MSCA, prompt: ["Password:", "Re-enter Password:"], answer: ["password\r", "password\r"] }

# device PKI debug when running the task
RP/0/RP0/CPU0:Jan 19 13:09:45.279 UTC: pki_cmd[67569]: pki_enroll ...
RP/0/RP0/CPU0:Jan 19 13:09:45.283 UTC: pki_cmd[67569]: Failed to enroll router cert, ERROR: Operation not permitted
RP/0/RP0/CPU0:Jan 19 13:09:45.283 UTC: pki_cmd[67569]: %SECURITY-PKI-6-ERR_1_PARAM : Failed to enroll router certificate.  

Manually executing the command with different passwords gives the same debug messages:

RP/0/RP0/CPU0:ialonso-XR9KV#crypto ca enroll MSCA
Thu Jan 19 13:01:18.510 UTC
RP/0/RP0/CPU0:Jan 19 13:01:18.654 UTC: pki_cmd[66736]: Keypair label for leaf cert not found. Trustpoint label: MSCA
% Start certificate enrollment ... 
% Create a challenge password. You will need to verbally provide this
  password to the CA Administrator in order to revoke your certificate.
% For security reasons your password will not be saved in the configuration.
% Please make a note of it.

Password: RP/0/RP0/CPU0:Jan 19 13:01:18.670 UTC: pki_cmd[66736]: pki_enroll ...

Re-enter Password: 
Passwords did not match. Aborted.
RP/0/RP0/CPU0:Jan 19 13:01:26.123 UTC: pki_cmd[66736]: Failed to enroll router cert, ERROR: Operation not permitted
RP/0/RP0/CPU0:Jan 19 13:01:26.123 UTC: pki_cmd[66736]: %SECURITY-PKI-6-ERR_1_PARAM : Failed to enroll router certificate.  

It's worked once, so I guess that the format of the prompts and answers is correct. Is there anything else that can be tried, or investigated to understand why it fails?

Here below the detailed output of the task to run the command "crypto ca enroll MSCA", confirming that the passwords provided to the prompts don't match:

<172.16.0.73> EXEC /bin/sh -c 'rm -f -r /home/ubuntu/.ansible/tmp/ansible-local-205094uu2oc_mj/ansible-tmp-1674149676.5798278-205345-106298447001606/ > /dev/null 2>&1 && sleep 0'
ok: [XR9KV01] => {
    "changed": false,
    "invocation": {
        "module_args": {
            "commands": [
                {
                    "answer": [
                        "password\r",
                        "password\r"
                    ],
                    "command": "crypto ca enroll MSCA",
                    "prompt": [
                        "Password:",
                        "Re-enter Password:"
                    ]
                }
            ],
            "interval": 1,
            "match": "all",
            "retries": 10,
            "wait_for": null
        }
    },
    "stdout": [
        "% Start certificate enrollment ... \n% Create a challenge password. You will need to verbally provide this\n  password to the CA Administrator in order to revoke your certificate.\n% For security reasons your password will not be saved in the configuration.\n% Please make a note of it.\n\nPassword: \nRe-enter Password: \nPasswords did not match. Aborted."
    ],
    "stdout_lines": [
        [
            "% Start certificate enrollment ... ",
            "% Create a challenge password. You will need to verbally provide this",
            "  password to the CA Administrator in order to revoke your certificate.",
            "% For security reasons your password will not be saved in the configuration.",
            "% Please make a note of it.",
            "",
            "Password: ",
            "Re-enter Password: ",
            "Passwords did not match. Aborted."
        ]
    ]
}
ashwini-mhatre commented 1 year ago

@ialonsob

If Its manually not working on device i think password which you are providing is wrong.

for task times out following is the solution: Increase command timeout value in ansible.cfg

example: [persistent_connection] command_timeout = 600

Please refer https://docs.ansible.com/ansible/latest/reference_appendices/config.html#persistent-command-timeout for more details

ialonsob commented 1 year ago

Thanks @ashwini-mhatre

Regarding the password, on the device it is working ok manually if the passwords are the same. With the ansible play, if I specify a password it fails always, with the error "Passwords did not match".

For the command timeout, I've increased it to 600 but still the result it's the same, please see the details below.

Confirmed behaviors

I've performed a few more tests and I have confirmed the following behaviors:

Here below the details of the two options.

Empty password in the ansible task

In this case, the router completes the command correctly and gets a certificate, however the ansible task times out.

Ansible task:

---

# Enroll with no password
- name: Enroll the IOS-XR Routers with the Intermediate CA
  hosts: ROUTERS
  gather_facts: False
  connection: local
  tasks:
    - name: Enroll with the Intermediate CA via SCEP
      cisco.iosxr.iosxr_command:
        commands:
          - {command: crypto ca enroll MSCA, prompt: ["Password:", "Re-enter Password:"], answer: ["\r", "\r"] }

Results:

ubuntu@ialonso-server02:~/ansible$ cat ansible.cfg 
[defaults]
remote_tmp = /tmp/tmp.ansible-${USER}/tmp
inventory = ./ansible_hosts
host_key_checking = False
interpreter_python = auto_silent
timeout = 300

[persistent_connection]
command_timeout = 600
ubuntu@ialonso-server02:~/ansible$ ansible-playbook play_enroll_xr9kv_nopassword.yml   

PLAY [Enroll the IOS-XR Routers with the Intermediate CA] ******************************************************************************************************************************************************************************

TASK [Enroll with the Intermediate CA via SCEP] ****************************************************************************************************************************************************************************************
fatal: [XR9KV02]: FAILED! => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python3"}, "changed": false, "msg": "command timeout triggered, timeout value is 600 secs.\nSee the timeout setting options in the Network Debug and Troubleshooting Guide."}
fatal: [XR9KV01]: FAILED! => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python3"}, "changed": false, "msg": "command timeout triggered, timeout value is 600 secs.\nSee the timeout setting options in the Network Debug and Troubleshooting Guide."}

PLAY RECAP *****************************************************************************************************************************************************************************************************************************
XR9KV01                    : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   
XR9KV02                    : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

Non-empty password in the ansible task

Manually, the command works ok on the device if the password and re-enter password are the same. With the ansible task configuration below, the task fails with the error "Passwords did not match. Aborted."

Ansible task:

---

# Enroll with password
- name: Enroll the IOS-XR Routers with the Intermediate CA
  hosts: ROUTERS
  gather_facts: False
  connection: local
  tasks:
    - name: Enroll with the Intermediate CA via SCEP
      cisco.iosxr.iosxr_command:
        commands:
          - {command: crypto ca enroll MSCA, prompt: ["Password:", "Re-enter Password:"], answer: ["password\r", "password\r"] }

Detailed ansible output:

ubuntu@ialonso-server02:~/ansible$ ansible-playbook -vvvv play_enroll_xr9kv_password.yml 
ansible-playbook 2.9.14
(..)
<172.16.0.34> EXEC /bin/sh -c 'rm -f -r /home/ubuntu/.ansible/tmp/ansible-local-211318oa925uq2/ansible-tmp-1674226470.3711483-211350-3089754630695/ > /dev/null 2>&1 && sleep 0'
ok: [XR9KV02] => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    },
    "changed": false,
    "invocation": {
        "module_args": {
            "commands": [
                {
                    "answer": [
                        "password\r",
                        "password\r"
                    ],
                    "command": "crypto ca enroll MSCA",
                    "prompt": [
                        "Password:",
                        "Re-enter Password:"
                    ]
                }
            ],
            "interval": 1,
            "match": "all",
            "retries": 10,
            "wait_for": null
        }
    },
    "stdout": [
        "% Start certificate enrollment ... \n% Create a challenge password. You will need to verbally provide this\n  password to the CA Administrator in order to revoke your certificate.\n% For security reasons your password will not be saved in the configuration.\n% Please make a note of it.\n\nPassword: \nRe-enter Password: \nPasswords did not match. Aborted."
    ],
    "stdout_lines": [
        [
            "% Start certificate enrollment ... ",
            "% Create a challenge password. You will need to verbally provide this",
            "  password to the CA Administrator in order to revoke your certificate.",
            "% For security reasons your password will not be saved in the configuration.",
            "% Please make a note of it.",
            "",
            "Password: ",
            "Re-enter Password: ",
            "Passwords did not match. Aborted."
        ]
    ]
}
(..)
ashwini-mhatre commented 1 year ago

@ialonsob Try following example it should work

- name: Enroll with the Intermediate CA via SCEP
      cisco.iosxr.iosxr_command:
          commands:
              - {command: key config-key password-encryption, prompt: [ "Enter old key :", "Enter new key :", "Enter confirm key :" ], answer: [test1234, test12345, test12345] , check_all: true}

Notes: do not give \r in your password give only password value add check_all flag. set it as True.

Let me know if it works for you.

ialonsob commented 1 year ago

Thanks @ashwini-mhatre

I've tried with your suggestion (no \r in the password, and enable check_all flag). The task works on the routers both with empty password and a password value, after executing the task the router has a certificate. However, the ansible task times out, even if the task completed correctly in the router. The only situation where the ansible task doesn't time out is if I enable "debug crypto pki messages" in the router. And it's only with this specific debug that the task doesn't time out. Here below an example run where the debug is enabled in XR9KV01 and not in XR9KV02. Both routers have the same configuration, and both obtain a certificate after running the task, however the ansible task has timed out on the router where the debug is not enabled.

    - name: Enroll with the Intermediate CA via SCEP
      cisco.iosxr.iosxr_command:
        commands:
          - {command: crypto ca enroll MSCA, prompt: ["Password:", "Re-enter Password:"], answer: ["password", "password"], check_all: true}

ubuntu@ialonso-server02:~/ansible$ ansible-playbook play_enroll_xr9kv_password.yml         

PLAY [Enroll the IOS-XR Routers with the Intermediate CA] ******************************************************************************************************************************************************************************************************

TASK [Enroll with the Intermediate CA via SCEP] ****************************************************************************************************************************************************************************************************************
ok: [XR9KV01]
fatal: [XR9KV02]: FAILED! => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python3"}, "changed": false, "msg": "command timeout triggered, timeout value is 60 secs.\nSee the timeout setting options in the Network Debug and Troubleshooting Guide."}

TASK [Verify the Router certificate] ***************************************************************************************************************************************************************************************************************************
ok: [XR9KV01]

PLAY RECAP *****************************************************************************************************************************************************************************************************************************************************
XR9KV01                    : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
XR9KV02                    : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

Is the ansible task expecting something from the router that is only provided if that specific debug is enabled? Thanks

ashwini-mhatre commented 1 year ago

@ialonsob try to increase command_timeout value [persistent_connection] command_timeout = 600

if still its times out may be device is expecting something else when password is set or its returning some value which is not handled in ansible

Also could you please try https://docs.ansible.com/ansible/latest/collections/ansible/netcommon/cli_command_module.html This example if it works for you need to debug more in iosxr_command module

- name: multiple prompt, multiple answer (mandatory check for all prompts)
  ansible.netcommon.cli_command:
    command: copy sftp sftp://user@host//user/test.img
    check_all: true
    prompt:
    - Confirm download operation
    - Password
    - Do you want to change that to the standby image
    answer:
    - y
    - <password>
    - y
ialonsob commented 1 year ago

Thanks @ashwini-mhatre Increasing the command_timeout value makes no difference. The command is correctly executed in the router within few seconds, and the ansible task times out even with a 10-minute command timeout value.

I've tried using the "cli_command" module and the behavior is the same. The command is correctly executed in the router but the ansible task times out. Also, if I enable that specific debug in the router, the ansible task does not time out - exactly the same behavior than with iosxr_command module.

Do you have any ideas or suggestions on how this could be further investigated?

BTW, I tried including the command "debug crypto pki messages" in the ansible play, but the task still times out. The only case so far where the ansible task doesn't timeout is if the debug is enabled manually by a user logged in the router.

Thanks

ashwini-mhatre commented 1 year ago

@ialonsob try following ansible configuration. ansible_connection: ansible.netcommon.network_cli ansible_network_cli_ssh_type: paramiko

or ansible_connection: ansible.netcommon.network_cli ansible_network_cli_ssh_type: libssh

please share me the output.Actually I am not able to reproduce this bug with same command that's why asking you so many debug logs. sorry for inconvieance

ashwini-mhatre commented 1 year ago

@ialonsob have you tried above configs?

ashwini-mhatre commented 1 year ago

@ialonsob any update?