search

ansible / ansible

Ansible is a radically simple IT automation platform that makes your applications and systems easier to deploy and maintain. Automate everything from code deployment to network configuration to cloud management, in a language that approaches plain English, using SSH, with no agents to install on remote systems. https://docs.ansible.com.
https://www.ansible.com/
GNU General Public License v3.0
62.98k stars 23.91k forks source link

Output gets truncated for certain show commands in latest devel #40683

Closed mikewiebe closed 6 years ago

mikewiebe commented 6 years ago
SUMMARY

The nxos_install_os module uses several of the show install all cli commands to gather information needed to complete successful upgrades.

It appears that updates have gone in over the past couple of days that are causing the output of these commands to be truncated.

This same test was passing 3 days ago.

MUST FIX FOR 2.6

ISSUE TYPE
COMPONENT NAME

nxos_install_os

ANSIBLE VERSION
ansible 2.6.0dev0 (rel260/nxos_install_os 0db1fe35a4) last updated 2018/05/24 14:03:08 (GMT -400)
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/Users/mwiebe/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/mwiebe/Projects/nxos_ansible/fix_ansible/lib/ansible
  executable location = /Users/mwiebe/Projects/nxos_ansible/fix_ansible/bin/ansible
  python version = 2.7.13 (default, Apr  4 2017, 08:47:57) [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.38)]
CONFIGURATION
DEFAULT_ROLES_PATH(env: ANSIBLE_ROLES_PATH) = [u'/Users/mwiebe/Projects/nxos_ansible/fix_ansible/test/integration/targets']
PARAMIKO_HOST_KEY_AUTO_ADD(env: ANSIBLE_PARAMIKO_HOST_KEY_AUTO_ADD) = True
PERSISTENT_COMMAND_TIMEOUT(env: ANSIBLE_PERSISTENT_COMMAND_TIMEOUT) = 1000
PERSISTENT_CONNECT_TIMEOUT(env: ANSIBLE_PERSISTENT_CONNECT_TIMEOUT) = 1000
OS / ENVIRONMENT

MacOS

STEPS TO REPRODUCE

I added some debug code to stop and grab output from the commands in question here:

This particular debug session was using connection plugin network_cli

https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/network/nxos/nxos.py#L162

(Epdb) list
154             """
155             if opts is None:
156                 opts = {}
157     
158             import epdb ; epdb.serve()
159  ->         connection = self._get_connection()
160     
161             msgs = []
162             try:
163                 responses = connection.edit_config(config)
164                 out = json.loads(responses)[1:-1]
(Epdb) config
['terminal dont-ask', 'show install all impact nxos nxos.7.0.3.I7.1.bin non-disruptive']
(Epdb) opts
{'ignore_timeout': True}
(Epdb) connection = self._get_connection()
(Epdb) connection
<ansible.module_utils.connection.Connection object at 0x106e87dd0>
(Epdb) msgs = []
(Epdb) responses = connection.edit_config(config)
(Epdb) responses
u'["Enter configuration commands, one per line. End with CNTL/Z.", "", "Installer will perform impact only check. Please wait. \\n\\nVerifying image bootflash:/nxos.7.0.3.I7.1.bin for boot variable \\"nxos\\".", "Verifying image type."]'
(Epdb) out = json.loads(responses)[1:-1]
(Epdb) out
[u'', u'Installer will perform impact only check. Please wait. \n\nVerifying image bootflash:/nxos.7.0.3.I7.1.bin for boot variable "nxos".']

The output is getting truncated.

The full output should contain the following:

n9k# show install all impact nxos nxos.7.0.3.I7.1.bin non-disruptive
Installer will perform impact only check. Please wait. 

Verifying image bootflash:/nxos.7.0.3.I7.1.bin for boot variable "nxos".
[####################] 100% -- SUCCESS

Verifying image type.
[####################] 100% -- SUCCESS

Preparing "bios" version info using image bootflash:/nxos.7.0.3.I7.1.bin.
[####################] 100% -- SUCCESS

Preparing "nxos" version info using image bootflash:/nxos.7.0.3.I7.1.bin.
[####################] 100% -- SUCCESS

Performing module support checks.
[####################] 100% -- SUCCESS

Notifying services about system upgrade.
[####################] 100% -- SUCCESS

Compatibility check is done:
Module  bootable          Impact  Install-type  Reason
------  --------  --------------  ------------  ------
     1       yes      disruptive         reset  Reset due to single supervisor
    22       yes      disruptive         reset  Reset due to single supervisor
    24       yes      disruptive         reset  Reset due to single supervisor
    26       yes      disruptive         reset  Reset due to single supervisor
    28       yes      disruptive         reset  Reset due to single supervisor
    29       yes      disruptive         reset  Reset due to single supervisor
    30       yes      disruptive         reset  Reset due to single supervisor

Images will be upgraded according to following table:
Module       Image                  Running-Version(pri:alt)           New-Version  Upg-Required
------  ----------  ----------------------------------------  --------------------  ------------
     1       lcn9k                               7.0(3)I6(1)           7.0(3)I7(1)           yes
     1        bios                       v01.48(00:v01.40(00             v01.48(00            no
    22       lcn9k                               7.0(3)I6(1)           7.0(3)I7(1)           yes
    22        bios                       v01.48(00:v01.42(00             v01.48(00            no
    24       lcn9k                               7.0(3)I6(1)           7.0(3)I7(1)           yes
    24        bios                       v01.48(00:v01.42(00             v01.48(00            no
    26       lcn9k                               7.0(3)I6(1)           7.0(3)I7(1)           yes
    26        bios                       v01.48(00:v01.42(00             v01.48(00            no
    28        nxos                               7.0(3)I6(1)           7.0(3)I7(1)           yes
    28        bios     v08.32(10/18/2016):v08.06(09/10/2014)    v08.32(10/18/2016)            no
    29       lcn9k                               7.0(3)I6(1)           7.0(3)I7(1)           yes
    29        bios                       v01.48(00:v01.42(00             v01.48(00            no
    30       lcn9k                               7.0(3)I6(1)           7.0(3)I7(1)           yes
    30        bios                       v01.48(00:v01.42(00             v01.48(00            no

n9k#
mikewiebe commented 6 years ago

NOTE: There are other commands that will likely be impacted by this bug, but this is a good one to start with.

mikewiebe commented 6 years ago

Same problem for ansible_connection=local

ansibot commented 6 years ago

cc @GGabriele @jedelman8 @privateip @rahushen @rcarrillocruz @trishnaguha click here for bot help

mikewiebe commented 6 years ago

Here is information for connection plugin httpapi

In this case, the output does get returned but the call to json.loads(responses)[1:-1] does not process the output correctly. Empty [] is the result.

(Epdb) list
158             connection = self._get_connection()
159     
160             msgs = []
161             try:
162                 import epdb ; epdb.serve()
163  ->             responses = connection.edit_config(config)
164                 out = json.loads(responses)[1:-1]
165                 msg = out
166             except ConnectionError as e:
167                 code = getattr(e, 'code', 1)
168                 message = getattr(e, 'err', e)
(Epdb) responses = connection.edit_config(config)
(Epdb) responses
u'["{}", "Installer will perform impact only check. Please wait. \\n\\nVerifying image bootflash:/nxos.7.0.3.I7.1.bin for boot variable \\"nxos\\".\\n[#                   ]   0%[####################] 100% -- SUCCESS\\n\\nVerifying image type.\\n[#                   ]   0%[####################] 100% -- SUCCESS\\n\\nPreparing \\"bios\\" version info using image bootflash:/nxos.7.0.3.I7.1.bin.\\n[#                   ]   0%[####################] 100% -- SUCCESS\\n\\nPreparing \\"nxos\\" version info using image bootflash:/nxos.7.0.3.I7.1.bin.\\n[#                   ]   0%[####################] 100% -- SUCCESS\\n\\nPerforming module support checks.\\n[####################] 100% -- SUCCESS\\n\\nNotifying services about system upgrade.\\n[####################] 100% -- SUCCESS\\n\\n\\n\\nCompatibility check is done:\\nModule  bootable          Impact  Install-type  Reason\\n------  --------  --------------  ------------  ------\\n     1       yes      disruptive         reset  Reset due to single supervisor\\n    22       yes      disruptive         reset  Reset due to single supervisor\\n    24       yes      disruptive         reset  Reset due to single supervisor\\n    26       yes      disruptive         reset  Reset due to single supervisor\\n    28       yes      disruptive         reset  Reset due to single supervisor\\n    29       yes      disruptive         reset  Reset due to single supervisor\\n    30       yes      disruptive         reset  Reset due to single supervisor\\n\\n\\n\\nImages will be upgraded according to following table:\\nModule       Image                  Running-Version(pri:alt)           New-Version  Upg-Required\\n------  ----------  ----------------------------------------  --------------------  ------------\\n     1       lcn9k                               7.0(3)I6(1)           7.0(3)I7(1)           yes\\n     1        bios                       v01.48(00:v01.40(00             v01.48(00            no\\n    22       lcn9k                               7.0(3)I6(1)           7.0(3)I7(1)           yes\\n    22        bios                       v01.48(00:v01.42(00             v01.48(00            no\\n    24       lcn9k                               7.0(3)I6(1)           7.0(3)I7(1)           yes\\n    24        bios                       v01.48(00:v01.42(00             v01.48(00            no\\n    26       lcn9k                               7.0(3)I6(1)           7.0(3)I7(1)           yes\\n    26        bios                       v01.48(00:v01.42(00             v01.48(00            no\\n    28        nxos                               7.0(3)I6(1)           7.0(3)I7(1)           yes\\n    28        bios     v08.32(10/18/2016):v08.06(09/10/2014)    v08.32(10/18/2016)            no\\n    29       lcn9k                               7.0(3)I6(1)           7.0(3)I7(1)           yes\\n    29        bios                       v01.48(00:v01.42(00             v01.48(00            no\\n    30       lcn9k                               7.0(3)I6(1)           7.0(3)I7(1)           yes\\n    30        bios                       v01.48(00:v01.42(00             v01.48(00            no"]'
(Epdb) out = json.loads(responses)[1:-1]
(Epdb) out
[]
mikewiebe commented 6 years ago

One additional data point. Transport nxapi still works without any issues.

TASK [nxos_install_os : Install OS image nxos.7.0.3.I7.1.bin] ************************************************************************************************************************************************************************************************
task path: /Users/mwiebe/Projects/nxos_ansible/fix_ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_system.yaml:2
<n9k.example.com> connection transport is nxapi
<n9k.example.com> ESTABLISH LOCAL CONNECTION FOR USER: mwiebe
<n9k.example.com> EXEC /bin/sh -c 'echo ~mwiebe && sleep 0'
<n9k.example.com> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/mwiebe/.ansible/tmp/ansible-tmp-1527192496.45-44896181855974 `" && echo ansible-tmp-1527192496.45-44896181855974="` echo /Users/mwiebe/.ansible/tmp/ansible-tmp-1527192496.45-44896181855974 `" ) && sleep 0'
Using module file /Users/mwiebe/Projects/nxos_ansible/fix_ansible/lib/ansible/modules/network/nxos/nxos_install_os.py
<n9k.example.com> PUT /Users/mwiebe/.ansible/tmp/ansible-local-649qTYBJ6/tmp3wFkMy TO /Users/mwiebe/.ansible/tmp/ansible-tmp-1527192496.45-44896181855974/nxos_install_os.py
<n9k.example.com> EXEC /bin/sh -c 'chmod u+x /Users/mwiebe/.ansible/tmp/ansible-tmp-1527192496.45-44896181855974/ /Users/mwiebe/.ansible/tmp/ansible-tmp-1527192496.45-44896181855974/nxos_install_os.py && sleep 0'
<n9k.example.com> EXEC /bin/sh -c '/Users/mwiebe/Virtualenvs/py2-ansible/bin/python /Users/mwiebe/.ansible/tmp/ansible-tmp-1527192496.45-44896181855974/nxos_install_os.py && sleep 0'
<n9k.example.com> EXEC /bin/sh -c 'rm -f -r /Users/mwiebe/.ansible/tmp/ansible-tmp-1527192496.45-44896181855974/ > /dev/null 2>&1 && sleep 0'
changed: [n9k.example.com] => {
    "changed": true, 
    "install_state": [
        "Compatibility check is done:", 
        "Module  bootable          Impact  Install-type  Reason", 
        "------  --------  --------------  ------------  ------", 
        "     1       yes      disruptive         reset  default upgrade is not hitless", 
        "    22       yes      disruptive         reset  default upgrade is not hitless", 
        "    24       yes      disruptive         reset  default upgrade is not hitless", 
        "    26       yes      disruptive         reset  default upgrade is not hitless", 
        "    28       yes      disruptive         reset  default upgrade is not hitless", 
        "    29       yes      disruptive         reset  default upgrade is not hitless", 
        "    30       yes      disruptive         reset  default upgrade is not hitless", 
        "Images will be upgraded according to following table:", 
        "Module       Image                  Running-Version(pri:alt)           New-Version  Upg-Required", 
        "------  ----------  ----------------------------------------  --------------------  ------------", 
        "     1       lcn9k                               7.0(3)I6(1)           7.0(3)I7(1)           yes", 
        "     1        bios                       v01.48(00:v01.40(00             v01.48(00            no", 
        "    22       lcn9k                               7.0(3)I6(1)           7.0(3)I7(1)           yes", 
        "    22        bios                       v01.48(00:v01.42(00             v01.48(00            no", 
        "    24       lcn9k                               7.0(3)I6(1)           7.0(3)I7(1)           yes", 
        "    24        bios                       v01.48(00:v01.42(00             v01.48(00            no", 
        "    26       lcn9k                               7.0(3)I6(1)           7.0(3)I7(1)           yes", 
        "    26        bios                       v01.48(00:v01.42(00             v01.48(00            no", 
        "    28        nxos                               7.0(3)I6(1)           7.0(3)I7(1)           yes", 
        "    28        bios     v08.32(10/18/2016):v08.06(09/10/2014)    v08.32(10/18/2016)            no", 
        "    29       lcn9k                               7.0(3)I6(1)           7.0(3)I7(1)           yes", 
        "    29        bios                       v01.48(00:v01.42(00             v01.48(00            no", 
        "    30       lcn9k                               7.0(3)I6(1)           7.0(3)I7(1)           yes", 
        "    30        bios                       v01.48(00:v01.42(00             v01.48(00            no", 
        "Module 1: Refreshing compact flash and upgrading bios/loader/bootrom.", 
        "Module 22: Refreshing compact flash and upgrading bios/loader/bootrom.", 
        "Module 24: Refreshing compact flash and upgrading bios/loader/bootrom.", 
        "Module 26: Refreshing compact flash and upgrading bios/loader/bootrom.", 
        "Module 28: Refreshing compact flash and upgrading bios/loader/bootrom.", 
        "Module 29: Refreshing compact flash and upgrading bios/loader/bootrom.", 
        "Module 30: Refreshing compact flash and upgrading bios/loader/bootrom."
    ], 
    "invocation": {
        "module_args": {
            "auth_pass": null, 
            "authorize": null, 
            "host": "n9k.example.com", 
            "issu": "desired", 
            "kickstart_image_file": null, 
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
            "port": 80, 
            "provider": {
                "auth_pass": null, 
                "authorize": null, 
                "host": "n9k.example.com", 
                "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
                "port": 80, 
                "ssh_keyfile": null, 
                "timeout": 1000, 
                "transport": "nxapi", 
                "use_proxy": true, 
                "use_ssl": false, 
                "username": "admin", 
                "validate_certs": true
            }, 
            "ssh_keyfile": null, 
            "system_image_file": "nxos.7.0.3.I7.1.bin", 
            "timeout": 1000, 
            "transport": "nxapi", 
            "url_password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
            "url_username": "admin", 
            "use_proxy": true, 
            "use_ssl": false, 
            "username": "admin", 
            "validate_certs": true
        }
    }
}

TASK [nxos_install_os : debug] *******************************************************************************************************************************************************************************************************************************
task path: /Users/mwiebe/Projects/nxos_ansible/fix_ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_system.yaml:10
ok: [n9k.example.com] => {
    "msg": " [u'Compatibility check is done:', u'Module  bootable          Impact  Install-type  Reason', u'------  --------  --------------  ------------  ------', u'     1       yes      disruptive         reset  default upgrade is not hitless', u'    22       yes      disruptive         reset  default upgrade is not hitless', u'    24       yes      disruptive         reset  default upgrade is not hitless', u'    26       yes      disruptive         reset  default upgrade is not hitless', u'    28       yes      disruptive         reset  default upgrade is not hitless', u'    29       yes      disruptive         reset  default upgrade is not hitless', u'    30       yes      disruptive         reset  default upgrade is not hitless', u'Images will be upgraded according to following table:', u'Module       Image                  Running-Version(pri:alt)           New-Version  Upg-Required', u'------  ----------  ----------------------------------------  --------------------  ------------', u'     1       lcn9k                               7.0(3)I6(1)           7.0(3)I7(1)           yes', u'     1        bios                       v01.48(00:v01.40(00             v01.48(00            no', u'    22       lcn9k                               7.0(3)I6(1)           7.0(3)I7(1)           yes', u'    22        bios                       v01.48(00:v01.42(00             v01.48(00            no', u'    24       lcn9k                               7.0(3)I6(1)           7.0(3)I7(1)           yes', u'    24        bios                       v01.48(00:v01.42(00             v01.48(00            no', u'    26       lcn9k                               7.0(3)I6(1)           7.0(3)I7(1)           yes', u'    26        bios                       v01.48(00:v01.42(00             v01.48(00            no', u'    28        nxos                               7.0(3)I6(1)           7.0(3)I7(1)           yes', u'    28        bios     v08.32(10/18/2016):v08.06(09/10/2014)    v08.32(10/18/2016)            no', u'    29       lcn9k                               7.0(3)I6(1)           7.0(3)I7(1)           yes', u'    29        bios                       v01.48(00:v01.42(00             v01.48(00            no', u'    30       lcn9k                               7.0(3)I6(1)           7.0(3)I7(1)           yes', u'    30        bios                       v01.48(00:v01.42(00             v01.48(00            no', u'Module 1: Refreshing compact flash and upgrading bios/loader/bootrom.', u'Module 22: Refreshing compact flash and upgrading bios/loader/bootrom.', u'Module 24: Refreshing compact flash and upgrading bios/loader/bootrom.', u'Module 26: Refreshing compact flash and upgrading bios/loader/bootrom.', u'Module 28: Refreshing compact flash and upgrading bios/loader/bootrom.', u'Module 29: Refreshing compact flash and upgrading bios/loader/bootrom.', u'Module 30: Refreshing compact flash and upgrading bios/loader/bootrom.']"
}

TASK [nxos_install_os : Wait for device to come back up with new image] **************************************************************************************************************************************************************************************
task path: /Users/mwiebe/Projects/nxos_ansible/fix_ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_system.yaml:12
<n9k.example.com> ESTABLISH LOCAL CONNECTION FOR USER: mwiebe
<n9k.example.com> EXEC /bin/sh -c 'echo ~mwiebe && sleep 0'
<n9k.example.com> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/mwiebe/.ansible/tmp/ansible-tmp-1527192707.4-90338391558389 `" && echo ansible-tmp-1527192707.4-90338391558389="` echo /Users/mwiebe/.ansible/tmp/ansible-tmp-1527192707.4-90338391558389 `" ) && sleep 0'
Using module file /Users/mwiebe/Projects/nxos_ansible/fix_ansible/lib/ansible/modules/utilities/logic/wait_for.py
<n9k.example.com> PUT /Users/mwiebe/.ansible/tmp/ansible-local-649qTYBJ6/tmpAIw9Qe TO /Users/mwiebe/.ansible/tmp/ansible-tmp-1527192707.4-90338391558389/wait_for.py
<n9k.example.com> EXEC /bin/sh -c 'chmod u+x /Users/mwiebe/.ansible/tmp/ansible-tmp-1527192707.4-90338391558389/ /Users/mwiebe/.ansible/tmp/ansible-tmp-1527192707.4-90338391558389/wait_for.py && sleep 0'
<n9k.example.com> EXEC /bin/sh -c '/Users/mwiebe/Virtualenvs/py2-ansible/bin/python /Users/mwiebe/.ansible/tmp/ansible-tmp-1527192707.4-90338391558389/wait_for.py && sleep 0'
<n9k.example.com> EXEC /bin/sh -c 'rm -f -r /Users/mwiebe/.ansible/tmp/ansible-tmp-1527192707.4-90338391558389/ > /dev/null 2>&1 && sleep 0'
ok: [n9k.example.com] => {
    "changed": false, 
    "elapsed": 186, 
    "invocation": {
        "module_args": {
            "active_connection_states": [
                "ESTABLISHED", 
                "FIN_WAIT1", 
                "FIN_WAIT2", 
                "SYN_RECV", 
                "SYN_SENT", 
                "TIME_WAIT"
            ], 
            "connect_timeout": 5, 
            "delay": 60, 
            "exclude_hosts": null, 
            "host": "n9k.example.com", 
            "msg": null, 
            "path": null, 
            "port": 22, 
            "search_regex": null, 
            "sleep": 1, 
            "state": "started", 
            "timeout": 500
        }
    }, 
    "path": null, 
    "port": 22, 
    "search_regex": null, 
    "state": "started"
}

TASK [nxos_install_os : debug] *******************************************************************************************************************************************************************************************************************************
task path: /Users/mwiebe/Projects/nxos_ansible/fix_ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_system.yaml:21
ok: [n9k.example.com] => {
    "msg": "Wait 5 mins to allow system to stabilize"
}

TASK [nxos_install_os : pause] *******************************************************************************************************************************************************************************************************************************
task path: /Users/mwiebe/Projects/nxos_ansible/fix_ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_system.yaml:23
Pausing for 300 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [n9k.example.com] => {
    "changed": false, 
    "delta": 300, 
    "echo": true, 
    "rc": 0, 
    "start": "2018-05-24 16:14:54.297931", 
    "stderr": "", 
    "stdout": "Paused for 300.0 seconds", 
    "stop": "2018-05-24 16:19:54.299903", 
    "user_input": ""
}
META: reset connection

TASK [nxos_install_os : Check installed OS for newly installed version {{ tv }}] *****************************************************************************************************************************************************************************
task path: /Users/mwiebe/Projects/nxos_ansible/fix_ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_os.yaml:21
<n9k.example.com> connection transport is nxapi
<n9k.example.com> ESTABLISH LOCAL CONNECTION FOR USER: mwiebe
<n9k.example.com> EXEC /bin/sh -c 'echo ~mwiebe && sleep 0'
<n9k.example.com> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/mwiebe/.ansible/tmp/ansible-tmp-1527193194.41-267519979468277 `" && echo ansible-tmp-1527193194.41-267519979468277="` echo /Users/mwiebe/.ansible/tmp/ansible-tmp-1527193194.41-267519979468277 `" ) && sleep 0'
Using module file /Users/mwiebe/Projects/nxos_ansible/fix_ansible/lib/ansible/modules/network/nxos/nxos_command.py
<n9k.example.com> PUT /Users/mwiebe/.ansible/tmp/ansible-local-649qTYBJ6/tmpyKMrGx TO /Users/mwiebe/.ansible/tmp/ansible-tmp-1527193194.41-267519979468277/nxos_command.py
<n9k.example.com> EXEC /bin/sh -c 'chmod u+x /Users/mwiebe/.ansible/tmp/ansible-tmp-1527193194.41-267519979468277/ /Users/mwiebe/.ansible/tmp/ansible-tmp-1527193194.41-267519979468277/nxos_command.py && sleep 0'
<n9k.example.com> EXEC /bin/sh -c '/Users/mwiebe/Virtualenvs/py2-ansible/bin/python /Users/mwiebe/.ansible/tmp/ansible-tmp-1527193194.41-267519979468277/nxos_command.py && sleep 0'
<n9k.example.com> EXEC /bin/sh -c 'rm -f -r /Users/mwiebe/.ansible/tmp/ansible-tmp-1527193194.41-267519979468277/ > /dev/null 2>&1 && sleep 0'
ok: [n9k.example.com] => {
    "changed": false, 
    "invocation": {
        "module_args": {
            "auth_pass": null, 
            "authorize": null, 
            "commands": [
                "show version | json"
            ], 
            "host": "n9k.example.com", 
            "interval": 1, 
            "match": "all", 
            "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
            "port": 80, 
            "provider": {
                "auth_pass": null, 
                "authorize": null, 
                "host": "n9k.example.com", 
                "password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
                "port": 80, 
                "ssh_keyfile": null, 
                "timeout": 1000, 
                "transport": "nxapi", 
                "use_proxy": true, 
                "use_ssl": false, 
                "username": "admin", 
                "validate_certs": true
            }, 
            "retries": 10, 
            "ssh_keyfile": null, 
            "timeout": 1000, 
            "transport": "nxapi", 
            "url_password": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
            "url_username": "admin", 
            "use_proxy": true, 
            "use_ssl": false, 
            "username": "admin", 
            "validate_certs": true, 
            "wait_for": null
        }
    }, 
    "stdout": [
        {
            "TABLE_package_list": {
                "ROW_package_list": {
                    "package_id": {}
                }
            }, 
            "bios_cmpl_time": "10/18/2016", 
            "bios_ver_str": "08.32", 
            "bootflash_size": 53298520, 
            "chassis_id": "Nexus9000 C9504 (4 Slot) Chassis", 
            "cpu_name": "Intel(R) Xeon(R) CPU E5-2403 0 @ 1.80GHz", 
            "header_str": "Cisco Nexus Operating System (NX-OS) Software\nTAC support: http://www.cisco.com/tac\nCopyright (C) 2002-2017, Cisco and/or its affiliates.\nAll rights reserved.\nThe copyrights to certain works contained in this software are\nowned by other third parties and used and distributed under their own\nlicenses, such as open source.  This software is provided \"as is,\" and unless\notherwise stated, there is no warranty, express or implied, including but not\nlimited to warranties of merchantability and fitness for a particular purpose.\nCertain components of this software are licensed under\nthe GNU General Public License (GPL) version 2.0 or \nGNU General Public License (GPL) version 3.0  or the GNU\nLesser General Public License (LGPL) Version 2.1 or \nLesser General Public License (LGPL) Version 2.0. \nA copy of each such license is available at\nhttp://www.opensource.org/licenses/gpl-2.0.php and\nhttp://opensource.org/licenses/gpl-3.0.html and\nhttp://www.opensource.org/licenses/lgpl-2.1.php and\nhttp://www.gnu.org/licenses/old-licenses/library.txt.\n", 
            "host_name": "n9k-109", 
            "kern_uptm_days": 0, 
            "kern_uptm_hrs": 0, 
            "kern_uptm_mins": 6, 
            "kern_uptm_secs": 45, 
            "kick_cmpl_time": " 8/31/2017 14:00:00", 
            "kick_file_name": "bootflash:///nxos.7.0.3.I7.1.bin", 
            "kick_tmstmp": "08/31/2017 22:29:32", 
            "kickstart_ver_str": "7.0(3)I7(1)", 
            "manufacturer": "Cisco Systems, Inc.", 
            "mem_type": "kB", 
            "memory": 16400800, 
            "module_id": "Supervisor Module", 
            "proc_board_id": "SAL1909A7VC", 
            "rr_ctime": " Thu May 24 20:11:57 2018\n", 
            "rr_reason": "Reset due to upgrade", 
            "rr_service": "", 
            "rr_sys_ver": "7.0(3)I6(1)", 
            "rr_usecs": 170577
        }
    ], 
    "stdout_lines": [
        {
            "TABLE_package_list": {
                "ROW_package_list": {
                    "package_id": {}
                }
            }, 
            "bios_cmpl_time": "10/18/2016", 
            "bios_ver_str": "08.32", 
            "bootflash_size": 53298520, 
            "chassis_id": "Nexus9000 C9504 (4 Slot) Chassis", 
            "cpu_name": "Intel(R) Xeon(R) CPU E5-2403 0 @ 1.80GHz", 
            "header_str": "Cisco Nexus Operating System (NX-OS) Software\nTAC support: http://www.cisco.com/tac\nCopyright (C) 2002-2017, Cisco and/or its affiliates.\nAll rights reserved.\nThe copyrights to certain works contained in this software are\nowned by other third parties and used and distributed under their own\nlicenses, such as open source.  This software is provided \"as is,\" and unless\notherwise stated, there is no warranty, express or implied, including but not\nlimited to warranties of merchantability and fitness for a particular purpose.\nCertain components of this software are licensed under\nthe GNU General Public License (GPL) version 2.0 or \nGNU General Public License (GPL) version 3.0  or the GNU\nLesser General Public License (LGPL) Version 2.1 or \nLesser General Public License (LGPL) Version 2.0. \nA copy of each such license is available at\nhttp://www.opensource.org/licenses/gpl-2.0.php and\nhttp://opensource.org/licenses/gpl-3.0.html and\nhttp://www.opensource.org/licenses/lgpl-2.1.php and\nhttp://www.gnu.org/licenses/old-licenses/library.txt.\n", 
            "host_name": "n9k-109", 
            "kern_uptm_days": 0, 
            "kern_uptm_hrs": 0, 
            "kern_uptm_mins": 6, 
            "kern_uptm_secs": 45, 
            "kick_cmpl_time": " 8/31/2017 14:00:00", 
            "kick_file_name": "bootflash:///nxos.7.0.3.I7.1.bin", 
            "kick_tmstmp": "08/31/2017 22:29:32", 
            "kickstart_ver_str": "7.0(3)I7(1)", 
            "manufacturer": "Cisco Systems, Inc.", 
            "mem_type": "kB", 
            "memory": 16400800, 
            "module_id": "Supervisor Module", 
            "proc_board_id": "SAL1909A7VC", 
            "rr_ctime": " Thu May 24 20:11:57 2018\n", 
            "rr_reason": "Reset due to upgrade", 
            "rr_service": "", 
            "rr_sys_ver": "7.0(3)I6(1)", 
            "rr_usecs": 170577
        }
    ]
}

TASK [nxos_install_os : debug] *******************************************************************************************************************************************************************************************************************************
task path: /Users/mwiebe/Projects/nxos_ansible/fix_ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_os.yaml:27
ok: [n9k.example.com] => {
    "msg": "Version detected 7.0(3)I7(1)"
}

TASK [nxos_install_os : debug] *******************************************************************************************************************************************************************************************************************************
task path: /Users/mwiebe/Projects/nxos_ansible/fix_ansible/test/integration/targets/nxos_install_os/tests/common/upgrade.yaml:57
ok: [n9k.example.com] => {
    "msg": "END connection=local nxos_os_install upgrade"
}
META: ran handlers
META: ran handlers

PLAY RECAP ***************************************************************************************************************************************************************************************************************************************************
n9k.example.com          : ok=44   changed=1    unreachable=0    failed=0   

(py2-ansible) mwiebe@MWIEBE-M-D2TB:~/Projects/nxos_ansible/fix_ansible/test/integration$
gundalow commented 6 years ago

Can we binary chop between a broken commit and working commit to work out what broke this?

saichint commented 6 years ago

I am seeing this in a different scenario. If we have large output for a show command and then we execute another show command after this, the 1st show command output is still shown (though truncated) in the 2nd command output. The 2nd show command output is simply ignored if the 1st command output is very large. Depending on the size of the command output, if we have 3rd, 4th etc. show commands, the problem is still the same and everything after the 1st command output is not coming. Even more weird is the fact that, if we have multiple tasks in a playbook, the show commands in the 2nd task are also being masked by the show output of 1st task. This is easily repeatable using idempotent tests.

trishnaguha commented 6 years ago

We figured out the commit which broke this https://github.com/ansible/ansible/commit/d829ff6993aa3a7e8123d2188ab04f4bfd88c1d0

trishnaguha commented 6 years ago

The commit https://github.com/ansible/ansible/pull/40940 is now being reverted as it was causing blocking bug https://github.com/ansible/ansible/issues/38639. But we have another commit https://github.com/ansible/ansible/pull/40960 merged in for this issue. Can you please test on devel and let us know?

mikewiebe commented 6 years ago

Quick update for the output getting truncated issue using the nxos_os_install module. I still have more testing to do, but things are looking better for connection local, network_cli and nxapi. The module still errors out using httpapi but for a different reason now.

ansible 2.7.0.dev0 (devel f426fcd9a6) last updated 2018/05/31 16:19:30 (GMT -400)
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/Users/mwiebe/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/mwiebe/Projects/nxos_ansible/fix_ansible/lib/ansible
  executable location = /Users/mwiebe/Projects/nxos_ansible/fix_ansible/bin/ansible
  python version = 2.7.13 (default, Apr  4 2017, 08:47:57) [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.38)]

Here is the error:

TASK [nxos_install_os : Install OS image nxos.7.0.3.I7.1.bin] *******************************************************************************************************************************************************************************************
task path: /Users/mwiebe/Projects/nxos_ansible/fix_ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_system.yaml:2
<n9k.example.com> loaded API plugin for network_os nxos
<n9k.example.com> attempting to start connection
<n9k.example.com> using connection plugin httpapi
<n9k.example.com> found existing local domain socket, using it!
<n9k.example.com> updating play_context for connection
<n9k.example.com> 
<n9k.example.com> local domain socket path is /Users/mwiebe/.ansible/pc/bcdb512db6
<n9k.example.com> ESTABLISH LOCAL CONNECTION FOR USER: mwiebe
<n9k.example.com> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/mwiebe/.ansible/tmp/ansible-local-7441oer9mi/ansible-tmp-1527800581.4-171879161901963 `" && echo ansible-tmp-1527800581.4-171879161901963="` echo /Users/mwiebe/.ansible/tmp/ansible-local-7441oer9mi/ansible-tmp-1527800581.4-171879161901963 `" ) && sleep 0'
Using module file /Users/mwiebe/Projects/nxos_ansible/fix_ansible/lib/ansible/modules/network/nxos/nxos_install_os.py
<n9k.example.com> PUT /Users/mwiebe/.ansible/tmp/ansible-local-7441oer9mi/tmpJ1NS5P TO /Users/mwiebe/.ansible/tmp/ansible-local-7441oer9mi/ansible-tmp-1527800581.4-171879161901963/nxos_install_os.py
<n9k.example.com> EXEC /bin/sh -c 'chmod u+x /Users/mwiebe/.ansible/tmp/ansible-local-7441oer9mi/ansible-tmp-1527800581.4-171879161901963/ /Users/mwiebe/.ansible/tmp/ansible-local-7441oer9mi/ansible-tmp-1527800581.4-171879161901963/nxos_install_os.py && sleep 0'
<n9k.example.com> EXEC /bin/sh -c '/Users/mwiebe/Virtualenvs/py2-ansible/bin/python /Users/mwiebe/.ansible/tmp/ansible-local-7441oer9mi/ansible-tmp-1527800581.4-171879161901963/nxos_install_os.py && sleep 0'
<n9k.example.com> EXEC /bin/sh -c 'rm -f -r /Users/mwiebe/.ansible/tmp/ansible-local-7441oer9mi/ansible-tmp-1527800581.4-171879161901963/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/var/folders/ww/538d42ys18l5tjyz0l1yk1rc0000gn/T/ansible__xPJcJ/ansible_module_nxos_install_os.py", line 545, in <module>
    main()
  File "/var/folders/ww/538d42ys18l5tjyz0l1yk1rc0000gn/T/ansible__xPJcJ/ansible_module_nxos_install_os.py", line 530, in main
    install_result = do_install_all(module, issu, sif, kick=kif)
  File "/var/folders/ww/538d42ys18l5tjyz0l1yk1rc0000gn/T/ansible__xPJcJ/ansible_module_nxos_install_os.py", line 460, in do_install_all
    impact_data = check_mode(module, issu, image, kick)
  File "/var/folders/ww/538d42ys18l5tjyz0l1yk1rc0000gn/T/ansible__xPJcJ/ansible_module_nxos_install_os.py", line 449, in check_mode
    data = check_mode_nextgen(module, issu, image, kick)
  File "/var/folders/ww/538d42ys18l5tjyz0l1yk1rc0000gn/T/ansible__xPJcJ/ansible_module_nxos_install_os.py", line 423, in check_mode_nextgen
    data = parse_show_install(load_config(module, commands, True, opts))
  File "/var/folders/ww/538d42ys18l5tjyz0l1yk1rc0000gn/T/ansible__xPJcJ/ansible_module_nxos_install_os.py", line 223, in parse_show_install
    ud['list_data'] = data.split('\n')
AttributeError: 'list' object has no attribute 'split'

fatal: [n9k.example.com]: FAILED! => {
    "changed": false, 
    "module_stderr": "Traceback (most recent call last):\n  File \"/var/folders/ww/538d42ys18l5tjyz0l1yk1rc0000gn/T/ansible__xPJcJ/ansible_module_nxos_install_os.py\", line 545, in <module>\n    main()\n  File \"/var/folders/ww/538d42ys18l5tjyz0l1yk1rc0000gn/T/ansible__xPJcJ/ansible_module_nxos_install_os.py\", line 530, in main\n    install_result = do_install_all(module, issu, sif, kick=kif)\n  File \"/var/folders/ww/538d42ys18l5tjyz0l1yk1rc0000gn/T/ansible__xPJcJ/ansible_module_nxos_install_os.py\", line 460, in do_install_all\n    impact_data = check_mode(module, issu, image, kick)\n  File \"/var/folders/ww/538d42ys18l5tjyz0l1yk1rc0000gn/T/ansible__xPJcJ/ansible_module_nxos_install_os.py\", line 449, in check_mode\n    data = check_mode_nextgen(module, issu, image, kick)\n  File \"/var/folders/ww/538d42ys18l5tjyz0l1yk1rc0000gn/T/ansible__xPJcJ/ansible_module_nxos_install_os.py\", line 423, in check_mode_nextgen\n    data = parse_show_install(load_config(module, commands, True, opts))\n  File \"/var/folders/ww/538d42ys18l5tjyz0l1yk1rc0000gn/T/ansible__xPJcJ/ansible_module_nxos_install_os.py\", line 223, in parse_show_install\n    ud['list_data'] = data.split('\\n')\nAttributeError: 'list' object has no attribute 'split'\n", 
    "module_stdout": "", 
    "msg": "MODULE FAILURE", 
    "rc": 1
}
    to retry, use: --limit @/Users/mwiebe/Projects/nxos_ansible/fix_ansible/test/integration/nxos.retry

PLAY RECAP **********************************************************************************************************************************************************************************************************************************************
n9k.example.com          : ok=44   changed=0    unreachable=0    failed=1
mikewiebe commented 6 years ago

Sadly.. the meta: reset_connection issue also seems to be back in the latest devel :(

META: reset connection

TASK [nxos_install_os : Check installed OS for newly installed version {{ tv }}] *******************************************************************************************************************************************************
task path: /Users/mwiebe/Projects/nxos_ansible/fix_ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_os.yaml:18
<n9k.example.com> attempting to start connection
<n9k.example.com> using connection plugin network_cli
<n9k.example.com> found existing local domain socket, using it!
<n9k.example.com> updating play_context for connection
<n9k.example.com> 
<n9k.example.com> local domain socket path is /Users/mwiebe/.ansible/pc/da511704a2
<n9k.example.com> ESTABLISH LOCAL CONNECTION FOR USER: mwiebe
<n9k.example.com> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /Users/mwiebe/.ansible/tmp/ansible-local-12318lQHJs6/ansible-tmp-1527802265.97-17083926370252 `" && echo ansible-tmp-1527802265.97-17083926370252="` echo /Users/mwiebe/.ansible/tmp/ansible-local-12318lQHJs6/ansible-tmp-1527802265.97-17083926370252 `" ) && sleep 0'
Using module file /Users/mwiebe/Projects/nxos_ansible/fix_ansible/lib/ansible/modules/network/nxos/nxos_command.py
<n9k.example.com> PUT /Users/mwiebe/.ansible/tmp/ansible-local-12318lQHJs6/tmpiHPi0k TO /Users/mwiebe/.ansible/tmp/ansible-local-12318lQHJs6/ansible-tmp-1527802265.97-17083926370252/nxos_command.py
<n9k.example.com> EXEC /bin/sh -c 'chmod u+x /Users/mwiebe/.ansible/tmp/ansible-local-12318lQHJs6/ansible-tmp-1527802265.97-17083926370252/ /Users/mwiebe/.ansible/tmp/ansible-local-12318lQHJs6/ansible-tmp-1527802265.97-17083926370252/nxos_command.py && sleep 0'
<n9k.example.com> EXEC /bin/sh -c '/Users/mwiebe/Virtualenvs/py2-ansible/bin/python /Users/mwiebe/.ansible/tmp/ansible-local-12318lQHJs6/ansible-tmp-1527802265.97-17083926370252/nxos_command.py && sleep 0'
<n9k.example.com> EXEC /bin/sh -c 'rm -f -r /Users/mwiebe/.ansible/tmp/ansible-local-12318lQHJs6/ansible-tmp-1527802265.97-17083926370252/ > /dev/null 2>&1 && sleep 0'
ok: [n9k.example.com] => {
    "changed": false, 
    "invocation": {
        "module_args": {
            "auth_pass": null, 
            "authorize": null, 
            "commands": [
                "show version | json"
            ], 
            "host": null, 
            "interval": 1, 
            "match": "all", 
            "password": null, 
            "port": null, 
            "provider": null, 
            "retries": 10, 
            "ssh_keyfile": null, 
            "timeout": null, 
            "transport": null, 
            "use_ssl": null, 
            "username": null, 
            "validate_certs": null, 
            "wait_for": null
        }
    }, 
    "stdout": [
        "None"
    ], 
    "stdout_lines": [
        [
            "None"
        ]
    ]
}

TASK [nxos_install_os : debug] *********************************************************************************************************************************************************************************************************
task path: /Users/mwiebe/Projects/nxos_ansible/fix_ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_os.yaml:23
fatal: [n9k.example.com]: FAILED! => {
    "msg": "The task includes an option with an undefined variable. The error was: 'list object' has no attribute 'kickstart_ver_str'\n\nThe error appears to have been in '/Users/mwiebe/Projects/nxos_ansible/fix_ansible/test/integration/targets/nxos_install_os/tasks/upgrade/install_os.yaml': line 23, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- debug: msg=\"Version detected {{ output['stdout_lines'][0]['kickstart_ver_str'] }}\"\n  ^ here\nWe could be wrong, but this one looks like it might be an issue with\nmissing quotes.  Always quote template expression brackets when they\nstart a value. For instance:\n\n    with_items:\n      - {{ foo }}\n\nShould be written as:\n\n    with_items:\n      - \"{{ foo }}\"\n"
}
    to retry, use: --limit @/Users/mwiebe/Projects/nxos_ansible/fix_ansible/test/integration/nxos.retry

PLAY RECAP *****************************************************************************************************************************************************************************************************************************
n9k.example.com          : ok=49   changed=1    unreachable=0    failed=1
mikewiebe commented 6 years ago

I believe we can consider this issue closed for the truncated output problem. I will file separate issues for meta: reset_connection and module nxos_install_os crash using connection httpapi

trishnaguha commented 6 years ago

@mikewiebe That sounds good to me.

trishnaguha commented 6 years ago

resolved_by_pr https://github.com/ansible/ansible/pull/40960