timeout [regression] Exception in multiple nxos_* modules when using SSH transport

[jean-christophe-manciot]

ISSUE TYPE

• Bug Report

  COMPONENT NAME

nxos_nxapi nxos_command nxos_template

ANSIBLE VERSION

ansible --version
ansible 2.3.0

CONFIGURATION

inventory = ./hosts roles_path = /home/actionmystique/Program-Files/Ubuntu/Ansible/git-Ansible/Roles/roles fact_caching = redis fact_caching_timeout = 86400

OS / ENVIRONMENT

• Local host: Ubuntu 16.10 4.8
• Target nodes:
  • NX-OSv 7.3(0)D1(1) (last release available in Cisco VIRL)
  • NX-OSv 7.2(0)

    SUMMARY

Running nxos_nxapi triggers an exception, whereas I can manually login into the target node with SSH.

STEPS TO REPRODUCE

Inventory ./hosts:

[nx_osv]
192.168.137.244 ansible_host=NX_OSv_1 mgt_interface=mgmt0
192.168.137.234 ansible_host=NX_OSv_2 mgt_interface=mgmt0

Structure passed as "provider": connections.ssh Defined in group_vars/nx_osv/connections.yml

connections:
        compute: local
...
        ssh:
          transport: cli 
          host: "{{ inventory_hostname }}"
          # ansible_port
          port: 22
          # ansible_user
          username: admin
          # ansible_ssh_pass
          password: xxxxxxxxxxx
          authorize: yes
          # enable_secret_password
          auth_pass: xxxxxxxxxx
          # private_key_file
          ssh_keyfile: "~/.ssh/id_rsa"
          version: 2
          timeout: 10
ansible_connection: "{{ connections.compute }}" 

Structures http & https Defined in group_vars/nx_osv/http.yml

http:
        # enable_http can be true / false
        service: true
        port: 80
https:
        # enable_https can be true / false
        service: false
        port: 10443

Role: nxos_init

- include_vars: "{{ role_path }}/../../group_vars/{{ hostvars[inventory_hostname].group_names[0] }}/connections.yml"
- include_vars: "{{ role_path }}/../../group_vars/{{ hostvars[inventory_hostname].group_names[0] }}/http.yml"

- name: Enable NX-API
  nxos_nxapi: 
      state=started 
      sandbox=false
      http={{ http.service }}
      http_port={{ http.port }}
      https={{ https.service }}
      https_port={{ https.port }}
      provider={{ connections.ssh }}
  register: return
...

Playbook: init.yml

- name: Initialize all NX-OS features & save facts
  hosts:
    - nx_os
    - nx_osv
  roles:
    - nxos_init

EXPECTED RESULTS

Successful nxos_nxapi

ACTUAL RESULTS

TASK [nxos_init : Enable NX-API] ***********************************************
task path: /home/actionmystique/Program-Files/Ubuntu/Ansible/git-Ansible/roles/nxos_init/tasks/main.yml:29
<NX_OSv_1> ESTABLISH LOCAL CONNECTION FOR USER: root
<NX_OSv_1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1470230351.99-196486228076050 `" && echo ansible-tmp-1470230351.99-196486228076050="` echo $HOME/.ansible/tmp/ansible-tmp-1470230351.99-196486228076050 `" ) && sleep 0'
<NX_OSv_2> ESTABLISH LOCAL CONNECTION FOR USER: root
<NX_OSv_2> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1470230351.99-268650786865216 `" && echo ansible-tmp-1470230351.99-268650786865216="` echo $HOME/.ansible/tmp/ansible-tmp-1470230351.99-268650786865216 `" ) && sleep 0'
<NX_OSv_2> PUT /tmp/tmpu2bixy TO /root/.ansible/tmp/ansible-tmp-1470230351.99-268650786865216/nxos_nxapi
<NX_OSv_1> PUT /tmp/tmpyyn3Hd TO /root/.ansible/tmp/ansible-tmp-1470230351.99-196486228076050/nxos_nxapi
<NX_OSv_2> EXEC /bin/sh -c 'LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python /root/.ansible/tmp/ansible-tmp-1470230351.99-268650786865216/nxos_nxapi; rm -rf "/root/.ansible/tmp/ansible-tmp-1470230351.99-268650786865216/" > /dev/null 2>&1 && sleep 0'
<NX_OSv_1> EXEC /bin/sh -c 'LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python /root/.ansible/tmp/ansible-tmp-1470230351.99-196486228076050/nxos_nxapi; rm -rf "/root/.ansible/tmp/ansible-tmp-1470230351.99-196486228076050/" > /dev/null 2>&1 && sleep 0'
An exception occurred during task execution. The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_4CyDsX/ansible_module_nxos_nxapi.py", line 257, in <module>
    main()
  File "/tmp/ansible_4CyDsX/ansible_module_nxos_nxapi.py", line 248, in main
    result = config_server(module)
  File "/tmp/ansible_4CyDsX/ansible_module_nxos_nxapi.py", line 156, in config_server
    nxapi_state = get_nxapi_state(module)
  File "/tmp/ansible_4CyDsX/ansible_module_nxos_nxapi.py", line 148, in get_nxapi_state
    features = module.execute(['show feature | grep nxapi'])[0]
  File "/tmp/ansible_4CyDsX/ansible_modlib.zip/ansible/module_utils/nxos.py", line 253, in execute
  File "/tmp/ansible_4CyDsX/ansible_modlib.zip/ansible/module_utils/nxos.py", line 230, in connect
  File "/tmp/ansible_4CyDsX/ansible_modlib.zip/ansible/module_utils/nxos.py", line 182, in connect
  File "/tmp/ansible_4CyDsX/ansible_modlib.zip/ansible/module_utils/shell.py", line 120, in open
  File "/tmp/ansible_4CyDsX/ansible_modlib.zip/ansible/module_utils/shell.py", line 129, in receive
  File "/usr/local/lib/python2.7/dist-packages/paramiko/channel.py", line 615, in recv
    raise socket.timeout()
socket.timeout
...

Troubleshooting

• Downgrading paramiko from 2.0.1 to 1.16 does not change anything
• When trying to manually ssh into the target:

# ssh admin@192.168.137.234
User Access Verification
Cisco NX-OS Software
Copyright (c) 2002-2016, Cisco Systems, Inc. All rights reserved.
NX-OSv software ("NX-OSv Software") and related documentation,
...
NX_OSv_2# exit
Connection to 192.168.137.234 closed.

[ansibot]

@jean-christophe-manciot we are unable to validate that "" is an existing ansible module.

We are expecting a component name that matches something that can be found in:

• http://docs.ansible.com/ansible/list_of_all_modules.html

Please revise the component name in the description. click here for bot help

[jean-christophe-manciot]

COMPONENT NAME

nxos_nxapi

[jean-christophe-manciot]

Same issue with nxos_command run with "show running-config".

[ansibot]

@privateip, @GGabriele, @gundalow, ping. This issue is waiting on your response. click here for bot help

[jean-christophe-manciot]

Same issue with nxos_template.

[jean-christophe-manciot]

No such issue with any ios_* modules.

[gundalow]

Reviewed, this needs fixing for 2.2

[ansibot]

@jedelman8, @GGabriele, @privateip, @gundalow, ping. This issue is still waiting on your response. click here for bot help

[ansibot]

@jedelman8, @GGabriele, @privateip, @gundalow, ping. This issue is still waiting on your response. click here for bot help

[ansibot]

@jedelman8, @GGabriele, @privateip, @gundalow, ping. This issue is still waiting on your response. click here for bot help

[ansibot]

@jedelman8, @GGabriele, @privateip, @gundalow, ping. This issue is still waiting on your response. click here for bot help

[jean-christophe-manciot]

@privateip With ansible 2.3.0 commit 20161111.3f785ee The issue described in this thread - accessing NX-OS devices - has been fixed for NX-API, not when using CLI/SSH transport, and there some cases where SSH is largely preferable to NX-API:

- name: Enable NX-API
  nxos_nxapi: 
      state: started 
      sandbox: "{{ sandbox }}"
      http: "{{ http.service }}"
      http_port: "{{ http.port }}"
      https: "{{ https.service }}"
      https_port: "{{ https.port }}"
      provider: "{{ connections.ssh }}"
  register: nxapi
  ignore_errors: yes

leads to:

TASK [nxos_init : Enable NX-API] ********************************************************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: socket.timeout
fatal: [NX_OSv_Spine_12]: FAILED! => {"changed": false, "failed": true, "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible__qOFX0/ansible_module_nxos_nxapi.py\", line 326, in <module>\n    main()\n  File \"/tmp/ansible__qOFX0/ansible_module_nxos_nxapi.py\", line 310, in main\n    instance = get_instance(module)\n  File \"/tmp/ansible__qOFX0/ansible_module_nxos_nxapi.py\", line 157, in get_instance\n    resp = module.cli('show nxapi', 'json')\n  File \"/tmp/ansible__qOFX0/ansible_modlib.zip/ansible/module_utils/network.py\", line 117, in cli\n  File \"/tmp/ansible__qOFX0/ansible_modlib.zip/ansible/module_utils/network.py\", line 148, in connect\n  File \"/tmp/ansible__qOFX0/ansible_modlib.zip/ansible/module_utils/nxos.py\", line 267, in connect\n  File \"/tmp/ansible__qOFX0/ansible_modlib.zip/ansible/module_utils/shell.py\", line 226, in connect\n  File \"/tmp/ansible__qOFX0/ansible_modlib.zip/ansible/module_utils/shell.py\", line 116, in open\n  File \"/tmp/ansible__qOFX0/ansible_modlib.zip/ansible/module_utils/shell.py\", line 128, in receive\n  File \"/usr/local/lib/python2.7/dist-packages/paramiko/channel.py\", line 615, in recv\n    raise socket.timeout()\nsocket.timeout\n", "module_stdout": "", "msg": "MODULE FAILURE"}

although:

ssh admin@172.21.100.12
...
NX_OSv_Spine_11# conf t
Enter configuration commands, one per line.  End with CNTL/Z.
NX_OSv_Spine_11(config)# feature nxapi 
NX_OSv_Spine_11(config)# 

and in ./hosts:

...
[spines]
NX_OSv_Spine_11 ansible_host=172.21.100.11 
NX_OSv_Spine_12 ansible_host=172.21.100.12 
...

Same issue with nxos_command:

- name: Fetching ARP_Table from the remote node
  nxos_command:
        provider: "{{ connections.ssh }}"
        commands:
          - "show ip arp vrf all"
  register: table
  ignore_errors: yes

leads to:

TASK [nxos_pull_tables : Fetching ARP_Table from the remote node] ***********************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: socket.timeout
fatal: [NX_OSv_Spine_11]: FAILED! => {"changed": false, "failed": true, "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_MPuLPC/ansible_module_nxos_command.py\", line 257, in <module>\n    main()\n  File \"/tmp/ansible_MPuLPC/ansible_module_nxos_command.py\", line 193, in main\n    supports_check_mode=True)\n  File \"/tmp/ansible_MPuLPC/ansible_modlib.zip/ansible/module_utils/network.py\", line 112, in __init__\n  File \"/tmp/ansible_MPuLPC/ansible_modlib.zip/ansible/module_utils/network.py\", line 148, in connect\n  File \"/tmp/ansible_MPuLPC/ansible_modlib.zip/ansible/module_utils/nxos.py\", line 267, in connect\n  File \"/tmp/ansible_MPuLPC/ansible_modlib.zip/ansible/module_utils/shell.py\", line 226, in connect\n  File \"/tmp/ansible_MPuLPC/ansible_modlib.zip/ansible/module_utils/shell.py\", line 116, in open\n  File \"/tmp/ansible_MPuLPC/ansible_modlib.zip/ansible/module_utils/shell.py\", line 128, in receive\n  File \"/usr/local/lib/python2.7/dist-packages/paramiko/channel.py\", line 615, in recv\n    raise socket.timeout()\nsocket.timeout\n", "module_stdout": "", "msg": "MODULE FAILURE"}

although:

ssh admin@172.21.100.11
...
NX_OSv_Spine_11# conf t
Enter configuration commands, one per line.  End with CNTL/Z.
NX_OSv_Spine_11(config)# exit
NX_OSv_Spine_11# sh ip arp vrf all

Flags: * - Adjacencies learnt on non-active FHRP router
       + - Adjacencies synced via CFSoE
       # - Adjacencies Throttled for Glean
       D - Static Adjacencies attached to down interface

IP ARP Table for all contexts
Total number of entries: 2
Address         Age       MAC Address     Interface
172.21.100.1    00:04:13  5254.009a.a68c  mgmt0           
172.21.100.12   00:03:22  0000.ab5a.0d00  mgmt0    

[gundalow]

Fix will go into stable-2.2

[ansibot]

@jedelman8, @GGabriele, @privateip, @gundalow, @qalthos, ping. This issue is still waiting on your response. click here for bot help

[ansibot]

This repository has been locked. All new issues and pull requests should be filed in https://github.com/ansible/ansible

Please read through the repomerge page in the dev guide. The guide contains links to tools which automatically move your issue or pull request to the ansible/ansible repo.