search

Juniper / ansible-junos-stdlib

Junos modules for Ansible
Apache License 2.0
305 stars 158 forks source link

juniper_junos_rpc - exceptions with large configurations #525

Closed jaakub closed 12 months ago

jaakub commented 4 years ago

Issue Type

Module Name

juniper_junos_rpc

juniper.device collection and Python libraries version

ansible 2.10.2
  config file = /home/myuser/.ansible.cfg
  configured module search path = ['/home/myuser/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/myuser/.venv_ansible/lib64/python3.6/site-packages/ansible
  executable location = /home/myuser/.venv_ansible/bin/ansible
  python version = 3.6.8 (default, Apr  2 2020, 13:34:55) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]

Package                      Version
---------------------------- ---------
ansible                      2.10.0
ansible-base                 2.10.2
junos-eznc                   2.5.3
jxmlease                     1.0.3
ncclient                     0.6.9
paramiko                     2.7.2

OS / Environment

Model: mx480
Junos: 17.3R3-S5.2
Configuration: 5936 lines
Task Success: FAIL
Model: mx960
Junos: 17.4R2-S7.3
Configuration: 74874 lines
Task Success: FAIL
Model: mx960
Junos: 17.4R2-S7.3
Configuration: 1097 lines
Task Success: SUCCESS
Model: mx240
Junos: 17.4R2-S7.3  
Configuration: 748 lines
Task Success: SUCCESS

Summary

When trying to run get-support-information and file-archive on MX nodes with larger configurations, more complex hardware (more interfaces etc), the task fails. When running on less complex nodes (less configuration lines and less interfaces), the RPCs complete OK. I increased the timeout to 20 minutes, but that doesn't help - it seems like a fundamental issue within code perhaps. I also tested again vMX running little config, and that completes successfully too, so it is clearly an issue with larger nodes.

Steps to reproduce

Run the below task against a device with multiple FPCs and large configuration.

- name: Execute RPC
  juniper_junos_rpc:
    rpcs:
      - get-support-information
      - file-archive
    kwargs:
      - {}
      - destination: "{{ junos_tgz }}"
        source: /var/log/*
        compress: True
    formats: text
    timeout: 1200
  register: junos_rpc

Expected results

TASK [Execute RPC] **************************************************************************************************************************************************************************************************
[WARNING]: The value "[{}, {'destination': '/var/tmp/my_device.example.com--13-10-2020--14-04.tgz', 'source': '/var/log/*', 'compress': True}]" (type list) was converted to ""[{}, {'destination': '/var/tmp/my_device.example.com--13-10-2020
--14-04.tgz', 'source': '/var/log/*', 'compress': True}]"" (type string). If this does not look like what you expect, quote the entire value to ensure it does not change.
[WARNING]: The value "22" (type int) was converted to "'22'" (type string). If this does not look like what you expect, quote the entire value to ensure it does not change.
ok: [my_device.example.com]

Actual results

TASK [Execute RPC] ******************************************************************************************************************************************************************************************************************
task path: /home/myuser/test-2.yml:29
redirecting (type: connection) ansible.builtin.netconf to ansible.netcommon.netconf
redirecting (type: netconf) ansible.builtin.junos to junipernetworks.junos.junos
redirecting (type: become) ansible.builtin.enable to ansible.netcommon.enable
<192.168.0.1> attempting to start connection
<192.168.0.1> using connection plugin ansible.netcommon.netconf
Found ansible-connection at path /home/myuser/.venv_ansible/bin/ansible-connection
<192.168.0.1> found existing local domain socket, using it!
<192.168.0.1> 
<192.168.0.1> local domain socket path is /home/myuser/.ansible/pc/8156450f7d
<192.168.0.1> ESTABLISH LOCAL CONNECTION FOR USER: myuser@domain.com
<192.168.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/myuser/.ansible/tmp/ansible-local-248375lpi59di `"&& mkdir "` echo /home/myuser/.ansible/tmp/ansible-local-248375lpi59di/ansible-tmp-1602593599.4167643-24932-16623649467381 `" && echo ansible-tmp-1602593599.4167643-24932-16623649467381="` echo /home/myuser/.ansible/tmp/ansible-local-248375lpi59di/ansible-tmp-1602593599.4167643-24932-16623649467381 `" ) && sleep 0'
Including module_utils file ansible/__init__.py
Including module_utils file ansible/module_utils/__init__.py
Including module_utils file ansible/module_utils/basic.py
Including module_utils file ansible/module_utils/_text.py
Including module_utils file ansible/module_utils/common/_collections_compat.py
Including module_utils file ansible/module_utils/common/__init__.py
Including module_utils file ansible/module_utils/common/_json_compat.py
Including module_utils file ansible/module_utils/common/_utils.py
Including module_utils file ansible/module_utils/common/file.py
Including module_utils file ansible/module_utils/common/parameters.py
Including module_utils file ansible/module_utils/common/collections.py
Including module_utils file ansible/module_utils/common/process.py
Including module_utils file ansible/module_utils/common/sys_info.py
Including module_utils file ansible/module_utils/common/text/converters.py
Including module_utils file ansible/module_utils/common/text/__init__.py
Including module_utils file ansible/module_utils/common/text/formatters.py
Including module_utils file ansible/module_utils/common/validation.py
Including module_utils file ansible/module_utils/common/warnings.py
Including module_utils file ansible/module_utils/compat/selectors.py
Including module_utils file ansible/module_utils/compat/__init__.py
Including module_utils file ansible/module_utils/compat/_selectors2.py
Including module_utils file ansible/module_utils/distro/__init__.py
Including module_utils file ansible/module_utils/distro/_distro.py
Including module_utils file ansible/module_utils/juniper_junos_common.py
Including module_utils file ansible/module_utils/parsing/convert_bool.py
Including module_utils file ansible/module_utils/parsing/__init__.py
Including module_utils file ansible/module_utils/pycompat24.py
Including module_utils file ansible/module_utils/six/__init__.py
Using module file /home/myuser/.ansible/roles/Juniper.junos/library/juniper_junos_rpc.py
<192.168.0.1> PUT /home/myuser/.ansible/tmp/ansible-local-248375lpi59di/tmpssqydpfo TO /home/myuser/.ansible/tmp/ansible-local-248375lpi59di/ansible-tmp-1602593599.4167643-24932-16623649467381/AnsiballZ_juniper_junos_rpc.py
<192.168.0.1> EXEC /bin/sh -c 'chmod u+x /home/myuser/.ansible/tmp/ansible-local-248375lpi59di/ansible-tmp-1602593599.4167643-24932-16623649467381/ /home/myuser/.ansible/tmp/ansible-local-248375lpi59di/ansible-tmp-1602593599.4167643-24932-16623649467381/AnsiballZ_juniper_junos_rpc.py && sleep 0'
<192.168.0.1> EXEC /bin/sh -c '/home/myuser/.venv_ansible/bin/python3 /home/myuser/.ansible/tmp/ansible-local-248375lpi59di/ansible-tmp-1602593599.4167643-24932-16623649467381/AnsiballZ_juniper_junos_rpc.py && sleep 0'
<192.168.0.1> EXEC /bin/sh -c 'rm -f -r /home/myuser/.ansible/tmp/ansible-local-248375lpi59di/ansible-tmp-1602593599.4167643-24932-16623649467381/ > /dev/null 2>&1 && sleep 0'
fatal: [lns1.hex]: FAILED! => {
    "changed": false,
    "module_stderr": "/home/myuser/.venv_ansible/lib64/python3.6/site-packages/jnpr/junos/device.py:837: RuntimeWarning: An unknown exception occurred - please report.\n  \"An unknown exception occurred - please report.\", RuntimeWarning\nTraceback (most recent call last):\n  File \"/home/myuser/.ansible/tmp/ansible-local-248375lpi59di/ansible-tmp-1602593599.4167643-24932-16623649467381/AnsiballZ_juniper_junos_rpc.py\", line 102, in <module>\n    _ansiballz_main()\n  File \"/home/myuser/.ansible/tmp/ansible-local-248375lpi59di/ansible-tmp-1602593599.4167643-24932-16623649467381/AnsiballZ_juniper_junos_rpc.py\", line 94, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/home/myuser/.ansible/tmp/ansible-local-248375lpi59di/ansible-tmp-1602593599.4167643-24932-16623649467381/AnsiballZ_juniper_junos_rpc.py\", line 40, in invoke_module\n    runpy.run_module(mod_name='ansible.modules.juniper_junos_rpc', init_globals=None, run_name='__main__', alter_sys=True)\n  File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code\n    mod_name, mod_spec, pkg_name, script_name)\n  File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_juniper_junos_rpc_payload_cvfqcx_k/ansible_juniper_junos_rpc_payload.zip/ansible/modules/juniper_junos_rpc.py\", line 646, in <module>\n  File \"/tmp/ansible_juniper_junos_rpc_payload_cvfqcx_k/ansible_juniper_junos_rpc_payload.zip/ansible/modules/juniper_junos_rpc.py\", line 567, in main\n  File \"/home/myuser/.venv_ansible/lib64/python3.6/site-packages/jnpr/junos/rpcmeta.py\", line 387, in __call__\n    return self._junos.execute(rpc_cmd, **kvargs)\n  File \"/home/myuser/.venv_ansible/lib64/python3.6/site-packages/jnpr/junos/decorators.py\", line 71, in wrapper\n    return function(*args, **kwargs)\n  File \"/home/myuser/.venv_ansible/lib64/python3.6/site-packages/jnpr/junos/decorators.py\", line 31, in wrapper\n    return function(*args, **kwargs)\n  File \"/home/myuser/.venv_ansible/lib64/python3.6/site-packages/jnpr/junos/device.py\", line 816, in execute\n    filter_xml=kvargs.get(\"filter_xml\"),\n  File \"/home/myuser/.venv_ansible/lib64/python3.6/site-packages/jnpr/junos/decorators.py\", line 117, in wrapper\n    rsp = function(self, *args, **kwargs)\n  File \"/home/myuser/.venv_ansible/lib64/python3.6/site-packages/jnpr/junos/device.py\", line 1419, in _rpc_reply\n    return self._conn.rpc(rpc_cmd_e, filter_xml)._NCElement__doc\n  File \"/home/myuser/.venv_ansible/lib64/python3.6/site-packages/ncclient/manager.py\", line 231, in execute\n    huge_tree=self._huge_tree).request(*args, **kwds)\n  File \"/home/myuser/.venv_ansible/lib64/python3.6/site-packages/ncclient/operations/third_party/juniper/rpc.py\", line 52, in request\n    return self._request(rpc)\n  File \"/home/myuser/.venv_ansible/lib64/python3.6/site-packages/ncclient/operations/rpc.py\", line 350, in _request\n    self._reply.parse()\n  File \"/home/myuser/.venv_ansible/lib64/python3.6/site-packages/ncclient/operations/rpc.py\", line 160, in parse\n    root = self._root = to_ele(self._raw, huge_tree=self._huge_tree) # The <rpc-reply> element\n  File \"/home/myuser/.venv_ansible/lib64/python3.6/site-packages/ncclient/xml_.py\", line 124, in to_ele\n    return x if etree.iselement(x) else etree.fromstring(x.encode('UTF-8'), parser=_get_parser(huge_tree))\n  File \"src/lxml/etree.pyx\", line 3237, in lxml.etree.fromstring\n  File \"src/lxml/parser.pxi\", line 1896, in lxml.etree._parseMemoryDocument\n  File \"src/lxml/parser.pxi\", line 1784, in lxml.etree._parseDoc\n  File \"src/lxml/parser.pxi\", line 1141, in lxml.etree._BaseParser._parseDoc\n  File \"src/lxml/parser.pxi\", line 615, in lxml.etree._ParserContext._handleParseResultDoc\n  File \"src/lxml/parser.pxi\", line 725, in lxml.etree._handleParseResult\n  File \"src/lxml/parser.pxi\", line 654, in lxml.etree._raiseParseError\n  File \"<string>\", line 422439\nlxml.etree.XMLSyntaxError: xmlSAX2Characters: huge text node, line 422439, column 18\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 1
}
rahkumar651991 commented 4 years ago

Hi @jacp9 - Can you share the netconf trace of the device for this scenario .

For reference - https://www.juniper.net/documentation/en_US/junos/topics/example/netconf-traceoptions-configuring.html

jaakub commented 4 years ago

Hi @rahkumar651991,

Sure - how can I securely provide that to you? It is quite a lot of files and obviously it contains the configuration.

netconf-ops.log                                                                                                         100% 1813KB   2.1MB/s   00:00    
netconf-ops.log.0.gz                                                                                                    100%   57KB   1.8MB/s   00:00    
netconf-ops.log.1.gz                                                                                                    100%  115KB   1.8MB/s   00:00    
netconf-ops.log.10.gz                                                                                                   100%  138KB   1.8MB/s   00:00    
netconf-ops.log.11.gz                                                                                                   100%   71KB   1.5MB/s   00:00    
netconf-ops.log.12.gz                                                                                                   100%  140KB   1.8MB/s   00:00    
netconf-ops.log.13.gz                                                                                                   100%  153KB   2.0MB/s   00:00    
netconf-ops.log.14.gz                                                                                                   100%  182KB   1.9MB/s   00:00    
netconf-ops.log.15.gz                                                                                                   100%  135KB   1.7MB/s   00:00    
netconf-ops.log.16.gz                                                                                                   100%   76KB   1.6MB/s   00:00    
netconf-ops.log.17.gz                                                                                                   100%   74KB   1.6MB/s   00:00    
netconf-ops.log.18.gz                                                                                                   100%  109KB   1.7MB/s   00:00    
netconf-ops.log.2.gz                                                                                                    100%   66KB   1.4MB/s   00:00    
netconf-ops.log.3.gz                                                                                                    100%   58KB   1.8MB/s   00:00    
netconf-ops.log.4.gz                                                                                                    100%  120KB   1.9MB/s   00:00    
netconf-ops.log.5.gz                                                                                                    100%  105KB   1.7MB/s   00:00    
netconf-ops.log.6.gz                                                                                                    100%  107KB   1.6MB/s   00:00    
netconf-ops.log.7.gz                                                                                                    100%  104KB   1.7MB/s   00:00    
netconf-ops.log.8.gz                                                                                                    100%  111KB   1.8MB/s   00:00    
netconf-ops.log.9.gz                                                                                                    100%  112KB   1.8MB/s   00:00
jaakub commented 4 years ago

@rahkumar651991 @vnitinv @ydnath - any ideas on the above? happy to work with you on that, it would be nice to be able to gather RSI/logs via Ansible and not manually.

rahkumar651991 commented 4 years ago

Hi @jacp9 - drop a mail to jnpr-community-netdev[at]juniper.net

spinoshi commented 2 years ago

Any update on this? It seems to be related to xml parsing and number of lines. https://github.com/ncclient/ncclient/issues/185 But I dont know where to set huge_tree=True

chidanandpujar commented 1 year ago

Hi ,

huge_tree is already supported as part of fix https://github.com/Juniper/ansible-junos-stdlib/pull/572. Could you please check following options and share the results .

---
- name: Test juniper.device.rpc module
  hosts: all
  connection: local
  gather_facts: no
  collections:
    - juniper.device

  tasks:
#################
   - name: Execute RPC
     rpc:
       rpcs:
         - get-support-information
         - file-archive
       kwargs:
         - {}
         - destination: "support_info"
           source: /var/log/*
           compress: True
       formats: text
       timeout: 1200
       huge_tree: True 
     register: junos_rpc
chidanandpujar commented 12 months ago

Hi @jaakub huge_tree is already supported as part of fix https://github.com/Juniper/ansible-junos-stdlib/pull/572. Please feel free to re-open the issue , if issue persists.

Thanks