search

vyos / vyos.vyos

Ansible Network Collection for VyOS
GNU General Public License v3.0
70 stars 64 forks source link

Fact discovery has trailing \u001b[m after each line/value #136

Closed sdwilsh closed 3 years ago

sdwilsh commented 3 years ago
SUMMARY

This looks similar to #65. I'm getting \u001b[m after every line or value when running vyos.vyos.vyos_facts, and this appears to break other things as well (I first came across this with an error when running vyos.vyos.vyos_firewall_rules and then proceeded to find a smaller test case to reproduce it with).

For example:

        ...
        "net_commits": [
            {
                "by": "ansible",
                "comment": null,
                "datetime": "2021-03-20 04:05:32 ",
                "revision": "0",
                "via": "cli\u001b[m"
            },
           ...
ISSUE TYPE
COMPONENT NAME

vyos.vyos.vyos_facts

ANSIBLE VERSION
ansible 2.9.6
  config file = /home/sdwilsh/ansible-playbooks/ansible.cfg
  configured module search path = ['/home/sdwilsh/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.8.5 (default, Jan 27 2021, 15:41:15) [GCC 9.3.0]
CONFIGURATION
DEFAULT_VAULT_PASSWORD_FILE(/home/sdwilsh/ansible-playbooks/ansible.cfg) = /home/sdwilsh/ansible-playbooks/.ansible_vault_password
INTERPRETER_PYTHON(/home/sdwilsh/ansible-playbooks/ansible.cfg) = auto
OS / ENVIRONMENT

Custom built vyos roughly equivalent to 1.3.0 RC1 (Build Commit ID: 3be456b18cf3ae)

STEPS TO REPRODUCE

ansible-playbook -i testing-inventory -l maccabeus.hogs.tswn.us get_facts.yml -u ansible -vvvv

---
- name: Get ansible facts
  hosts: vyos
  tasks:
    - vyos.vyos.vyos_facts:
        gather_subset: all
    - debug:
        msg: "The hostname is {{ ansible_net_hostname }} and the OS is {{ ansible_net_version }}"
EXPECTED RESULTS
    "msg": "The hostname is maccabeus.hogs.tswn.us and the OS is VyOS 1.3.0-rc1"
ACTUAL RESULTS
ansible-playbook 2.9.6
  config file = /home/sdwilsh/ansible-playbooks/ansible.cfg
  configured module search path = ['/home/sdwilsh/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  executable location = /usr/bin/ansible-playbook
  python version = 3.8.5 (default, Jan 27 2021, 15:41:15) [GCC 9.3.0]
Using /home/sdwilsh/ansible-playbooks/ansible.cfg as config file
setting up inventory plugins
host_list declined parsing /home/sdwilsh/ansible-playbooks/testing-inventory as it did not pass its verify_file() method
script declined parsing /home/sdwilsh/ansible-playbooks/testing-inventory as it did not pass its verify_file() method
auto declined parsing /home/sdwilsh/ansible-playbooks/testing-inventory as it did not pass its verify_file() method
Parsed /home/sdwilsh/ansible-playbooks/testing-inventory inventory source with ini plugin
Loading callback plugin default of type stdout, v2.0 from /usr/lib/python3/dist-packages/ansible/plugins/callback/default.py

PLAYBOOK: get_facts.yml ***************************************************************************************************************************************************************************************
Positional arguments: get_facts.yml
verbosity: 4
remote_user: ansible
connection: smart
timeout: 10
become_method: sudo
tags: ('all',)
inventory: ('/home/sdwilsh/ansible-playbooks/testing-inventory',)
subset: maccabeus.hogs.tswn.us
forks: 5
1 plays in get_facts.yml

PLAY [Get ansible facts] **************************************************************************************************************************************************************************************

TASK [Gathering Facts] ****************************************************************************************************************************************************************************************
task path: /home/sdwilsh/ansible-playbooks/get_facts.yml:2
<10.10.0.1> attempting to start connection
<10.10.0.1> using connection plugin network_cli
<10.10.0.1> local domain socket does not exist, starting it
<10.10.0.1> control socket path is /home/sdwilsh/.ansible/pc/79828898f7
<10.10.0.1> local domain socket listeners started successfully
<10.10.0.1> loaded cliconf plugin vyos from path /usr/lib/python3/dist-packages/ansible/plugins/cliconf/vyos.py for network_os vyos
<10.10.0.1> 
<10.10.0.1> local domain socket path is /home/sdwilsh/.ansible/pc/79828898f7
<10.10.0.1> ESTABLISH LOCAL CONNECTION FOR USER: sdwilsh
<10.10.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/sdwilsh/.ansible/tmp/ansible-local-268907f4_1n_d8/ansible-tmp-1616219810.1474397-58028577226599 `" && echo ansible-tmp-1616219810.1474397-58028577226599="` echo /home/sdwilsh/.ansible/tmp/ansible-local-268907f4_1n_d8/ansible-tmp-1616219810.1474397-58028577226599 `" ) && sleep 0'
<maccabeus.hogs.tswn.us> Attempting python interpreter discovery
<10.10.0.1> EXEC /bin/sh -c 'echo PLATFORM; uname; echo FOUND; command -v '"'"'/usr/bin/python'"'"'; command -v '"'"'python3.7'"'"'; command -v '"'"'python3.6'"'"'; command -v '"'"'python3.5'"'"'; command -v '"'"'python2.7'"'"'; command -v '"'"'python2.6'"'"'; command -v '"'"'/usr/libexec/platform-python'"'"'; command -v '"'"'/usr/bin/python3'"'"'; command -v '"'"'python'"'"'; echo ENDFOUND && sleep 0'
<10.10.0.1> EXEC /bin/sh -c '/usr/bin/python3 && sleep 0'
Using module file /usr/lib/python3/dist-packages/ansible/modules/network/vyos/vyos_facts.py
<10.10.0.1> PUT /home/sdwilsh/.ansible/tmp/ansible-local-268907f4_1n_d8/tmp73nsdxur TO /home/sdwilsh/.ansible/tmp/ansible-local-268907f4_1n_d8/ansible-tmp-1616219810.1474397-58028577226599/AnsiballZ_vyos_facts.py
<10.10.0.1> EXEC /bin/sh -c 'chmod u+x /home/sdwilsh/.ansible/tmp/ansible-local-268907f4_1n_d8/ansible-tmp-1616219810.1474397-58028577226599/ /home/sdwilsh/.ansible/tmp/ansible-local-268907f4_1n_d8/ansible-tmp-1616219810.1474397-58028577226599/AnsiballZ_vyos_facts.py && sleep 0'
<10.10.0.1> EXEC /bin/sh -c '/usr/bin/python3 /home/sdwilsh/.ansible/tmp/ansible-local-268907f4_1n_d8/ansible-tmp-1616219810.1474397-58028577226599/AnsiballZ_vyos_facts.py && sleep 0'
<10.10.0.1> EXEC /bin/sh -c 'rm -f -r /home/sdwilsh/.ansible/tmp/ansible-local-268907f4_1n_d8/ansible-tmp-1616219810.1474397-58028577226599/ > /dev/null 2>&1 && sleep 0'
ok: [maccabeus.hogs.tswn.us]
META: ran handlers

TASK [vyos.vyos.vyos_facts] ***********************************************************************************************************************************************************************************
task path: /home/sdwilsh/ansible-playbooks/get_facts.yml:5
<10.10.0.1> attempting to start connection
<10.10.0.1> using connection plugin network_cli
<10.10.0.1> found existing local domain socket, using it!
<10.10.0.1> ssh connection done, setting terminal
<10.10.0.1> loaded terminal plugin for network_os vyos
<10.10.0.1> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<10.10.0.1> firing event: on_open_shell()
<10.10.0.1> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<10.10.0.1> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<10.10.0.1> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<10.10.0.1> ssh connection has completed successfully
<10.10.0.1> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<10.10.0.1> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<10.10.0.1> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<10.10.0.1> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<10.10.0.1> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<10.10.0.1> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<10.10.0.1> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<10.10.0.1> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<10.10.0.1> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<10.10.0.1> updating play_context for connection
<10.10.0.1> 
<10.10.0.1> local domain socket path is /home/sdwilsh/.ansible/pc/79828898f7
<10.10.0.1> ESTABLISH LOCAL CONNECTION FOR USER: sdwilsh
<10.10.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/sdwilsh/.ansible/tmp/ansible-local-268907f4_1n_d8/ansible-tmp-1616219814.6760836-125456008838287 `" && echo ansible-tmp-1616219814.6760836-125456008838287="` echo /home/sdwilsh/.ansible/tmp/ansible-local-268907f4_1n_d8/ansible-tmp-1616219814.6760836-125456008838287 `" ) && sleep 0'
Using module file /home/sdwilsh/.ansible/collections/ansible_collections/vyos/vyos/plugins/modules/vyos_facts.py
<10.10.0.1> PUT /home/sdwilsh/.ansible/tmp/ansible-local-268907f4_1n_d8/tmptmcr0blm TO /home/sdwilsh/.ansible/tmp/ansible-local-268907f4_1n_d8/ansible-tmp-1616219814.6760836-125456008838287/AnsiballZ_vyos_facts.py
<10.10.0.1> EXEC /bin/sh -c 'chmod u+x /home/sdwilsh/.ansible/tmp/ansible-local-268907f4_1n_d8/ansible-tmp-1616219814.6760836-125456008838287/ /home/sdwilsh/.ansible/tmp/ansible-local-268907f4_1n_d8/ansible-tmp-1616219814.6760836-125456008838287/AnsiballZ_vyos_facts.py && sleep 0'
<10.10.0.1> EXEC /bin/sh -c '/usr/bin/python3 /home/sdwilsh/.ansible/tmp/ansible-local-268907f4_1n_d8/ansible-tmp-1616219814.6760836-125456008838287/AnsiballZ_vyos_facts.py && sleep 0'
<10.10.0.1> EXEC /bin/sh -c 'rm -f -r /home/sdwilsh/.ansible/tmp/ansible-local-268907f4_1n_d8/ansible-tmp-1616219814.6760836-125456008838287/ > /dev/null 2>&1 && sleep 0'
ok: [maccabeus.hogs.tswn.us] => {
    "ansible_facts": {
        "ansible_net_api": "cliconf",
        "ansible_net_commits": [
            {
                "by": "ansible",
                "comment": null,
                "datetime": "2021-03-20 04:05:32 ",
                "revision": "0",
                "via": "cli\u001b[m"
            },
            {
                "by": "ansible",
                "comment": null,
                "datetime": "2021-03-20 04:05:30 ",
                "revision": "1",
                "via": "cli\u001b[m"
            },
            {
                "by": "ansible",
                "comment": null,
                "datetime": "2021-03-20 03:58:32 ",
                "revision": "2",
                "via": "cli\u001b[m"
            },
            {
                "by": "vyos",
                "comment": null,
                "datetime": "2021-03-20 03:50:30 ",
                "revision": "3",
                "via": "cli\u001b[m"
            },
            {
                "by": "vyos",
                "comment": null,
                "datetime": "2021-03-20 03:47:25 ",
                "revision": "4",
                "via": "cli\u001b[m"
            },
            {
                "by": "vyos",
                "comment": null,
                "datetime": "2021-03-20 03:22:49 ",
                "revision": "5",
                "via": "cli\u001b[m"
            },
            {
                "by": "vyos",
                "comment": null,
                "datetime": "2021-03-20 03:20:59 ",
                "revision": "6",
                "via": "cli\u001b[m"
            },
            {
                "by": "vyos",
                "comment": null,
                "datetime": "2021-03-19 04:26:31 ",
                "revision": "7",
                "via": "cli\u001b[m"
            },
            {
                "by": "vyos",
                "comment": null,
                "datetime": "2021-03-19 04:18:34 ",
                "revision": "8",
                "via": "cli\u001b[m"
            },
            {
                "by": "vyos",
                "comment": null,
                "datetime": "2021-03-19 03:37:57 ",
                "revision": "9",
                "via": "cli\u001b[m"
            },
            {
                "by": "vyos",
                "comment": null,
                "datetime": "2021-03-19 03:31:44 ",
                "revision": "10",
                "via": "cli\u001b[m"
            },
            {
                "by": "vyos",
                "comment": null,
                "datetime": "2021-03-19 01:18:57 ",
                "revision": "11",
                "via": "cli\u001b[m"
            },
            {
                "by": "vyos",
                "comment": null,
                "datetime": "2021-03-19 01:06:16 ",
                "revision": "12",
                "via": "cli\u001b[m"
            },
            {
                "by": "vyos",
                "comment": null,
                "datetime": "2021-03-19 01:04:11 ",
                "revision": "13",
                "via": "cli\u001b[m"
            },
            {
                "by": "root",
                "comment": null,
                "datetime": "2021-03-19 00:59:37 ",
                "revision": "14",
                "via": "vyos-boot-config-loader\u001b[m"
            }
        ],
        "ansible_net_config": [
            "set firewall all-ping 'enable'\u001b[m\nset firewall broadcast-ping 'disable'\u001b[m\nset firewall config-trap 'disable'\u001b[m\nset firewall ipv6-receive-redirects 'disable'\u001b[m\nset firewall ipv6-src-route 'disable'\u001b[m\nset firewall ip-src-route 'disable'\u001b[m\nset firewall log-martians 'enable'\u001b[m\nset firewall name WAN-local default-action 'drop'\u001b[m\nset firewall name WAN-local rule 1 action 'accept'\u001b[m\nset firewall name WAN-local rule 1 state established 'enable'\u001b[m\nset firewall name WAN-local rule 1 state related 'enable'\u001b[m\nset firewall name WAN-local rule 2 action 'drop'\u001b[m\nset firewall name WAN-local rule 2 state invalid 'enable'\u001b[m\nset firewall name local-WAN default-action 'accept'\u001b[m\nset firewall receive-redirects 'disable'\u001b[m\nset firewall send-redirects 'enable'\u001b[m\nset firewall source-validation 'disable'\u001b[m\nset firewall syn-cookies 'enable'\u001b[m\nset firewall twa-hazards-protection 'disable'\u001b[m\nset interfaces ethernet eth0 hw-id '00:15:5d:01:c2:05'\u001b[m\nset interfaces ethernet eth0 vif 10 firewall in name 'local-WAN'\u001b[m\nset interfaces ethernet eth0 vif 10 firewall local name 'local-WAN'\u001b[m\nset interfaces ethernet eth0 vif 10 firewall out name 'local-WAN'\u001b[m\nset interfaces ethernet eth0 vif 201 description 'CenturyLink PPPoE'\u001b[m\nset interfaces ethernet eth1 hw-id '00:15:5d:01:c2:06'\u001b[m\nset interfaces ethernet eth1 vif 10 address '10.10.0.1/24'\u001b[m\nset interfaces loopback lo\u001b[m\nset interfaces pppoe pppoe0 authentication password 'vyos'\u001b[m\nset interfaces pppoe pppoe0 authentication user 'vyos'\u001b[m\nset interfaces pppoe pppoe0 default-route 'auto'\u001b[m\nset interfaces pppoe pppoe0 firewall in name 'local-WAN'\u001b[m\nset interfaces pppoe pppoe0 firewall local name 'local-WAN'\u001b[m\nset interfaces pppoe pppoe0 firewall out name 'local-WAN'\u001b[m\nset interfaces pppoe pppoe0 mtu '1492'\u001b[m\nset interfaces pppoe pppoe0 source-interface 'eth0.201'\u001b[m\nset service dhcp-server shared-network-name MAN authoritative\u001b[m\nset service dhcp-server shared-network-name MAN subnet 10.10.0.0/24 default-router '10.10.0.1'\u001b[m\nset service dhcp-server shared-network-name MAN subnet 10.10.0.0/24 range 0 start '10.10.0.200'\u001b[m\nset service dhcp-server shared-network-name MAN subnet 10.10.0.0/24 range 0 stop '10.10.0.250'\u001b[m\nset service ssh listen-address '10.10.0.1'\u001b[m\nset system config-management commit-revisions '100'\u001b[m\nset system console device ttyS0 speed '115200'\u001b[m\nset system domain-search domain 'hogs.tswn.us'\u001b[m\nset system host-name 'maccabeus.hogs.tswn.us'\u001b[m\nset system login user ansible authentication public-keys ansbile@sdwilsh-dev key 'AAAAC3NzaC1lZDI1NTE5AAAAIMiRn+dJyIkJ22qrkuMlNC33xxS7VUwkYRY/55Wf4ryq'\u001b[m\nset system login user ansible authentication public-keys ansbile@sdwilsh-dev type 'ssh-ed25519'\u001b[m\nset system login user ansible full-name 'ansible'\u001b[m\nset system login user vyos authentication encrypted-password '$6$GoYinRgTWrilPn$NNGRoUpms08mgTXAx3LxQG4TOh7462VdxuC9SIBjNYdlJ2lpjwZc5eplhGIfEu668r9ACZ4Y5LSXwWpgq8hFG/'\u001b[m\nset system login user vyos authentication plaintext-password ''\u001b[m\nset system name-server '10.117.0.3'\u001b[m\nset system name-server '10.117.0.4'\u001b[m\nset system name-server 'fd36:3eb3:43b0:75::3'\u001b[m\nset system name-server 'fd36:3eb3:43b0:75::4'\u001b[m\nset system ntp server 0.pool.ntp.org\u001b[m\nset system ntp server 1.pool.ntp.org\u001b[m\nset system ntp server 2.pool.ntp.org\u001b[m\nset system syslog global facility all level 'info'\u001b[m\nset system syslog global facility protocols level 'debug'\u001b[m",
            "0   2021-03-20 04:05:32 by ansible via cli\u001b[m\n1   2021-03-20 04:05:30 by ansible via cli\u001b[m\n2   2021-03-20 03:58:32 by ansible via cli\u001b[m\n3   2021-03-20 03:50:30 by vyos via cli\u001b[m\n4   2021-03-20 03:47:25 by vyos via cli\u001b[m\n5   2021-03-20 03:22:49 by vyos via cli\u001b[m\n6   2021-03-20 03:20:59 by vyos via cli\u001b[m\n7   2021-03-19 04:26:31 by vyos via cli\u001b[m\n8   2021-03-19 04:18:34 by vyos via cli\u001b[m\n9   2021-03-19 03:37:57 by vyos via cli\u001b[m\n10  2021-03-19 03:31:44 by vyos via cli\u001b[m\n11  2021-03-19 01:18:57 by vyos via cli\u001b[m\n12  2021-03-19 01:06:16 by vyos via cli\u001b[m\n13  2021-03-19 01:04:11 by vyos via cli\u001b[m\n14  2021-03-19 00:59:37 by root via vyos-boot-config-loader\u001b[m\n15  2021-03-19 00:59:36 by root via init\u001b[m"
        ],
        "ansible_net_gather_network_resources": [],
        "ansible_net_gather_subset": [
            "config",
            "neighbors",
            "default"
        ],
        "ansible_net_hostname": "maccabeus.hogs.tswn.us\u001b[m",
        "ansible_net_neighbors": {},
        "ansible_net_python_version": "3.8.5",
        "ansible_net_serialnum": null,
        "ansible_net_system": "vyos",
        "ansible_net_version": "VyOS 1.3.0-rc1\u001b[m",
        "ansible_network_resources": {}
    },
    "changed": false,
    "invocation": {
        "module_args": {
            "gather_network_resources": null,
            "gather_subset": [
                "all"
            ],
            "provider": null
        }
    }
}

TASK [debug] **************************************************************************************************************************************************************************************************
task path: /home/sdwilsh/ansible-playbooks/get_facts.yml:7
<10.10.0.1> attempting to start connection
<10.10.0.1> using connection plugin network_cli
<10.10.0.1> found existing local domain socket, using it!
<10.10.0.1> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<10.10.0.1> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<10.10.0.1> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<10.10.0.1> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<10.10.0.1> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<10.10.0.1> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<10.10.0.1> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<10.10.0.1> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<10.10.0.1> Response received, triggered 'persistent_buffer_read_timeout' timer of 0.1 seconds
<10.10.0.1> updating play_context for connection
<10.10.0.1> 
<10.10.0.1> local domain socket path is /home/sdwilsh/.ansible/pc/79828898f7
ok: [maccabeus.hogs.tswn.us] => {
    "msg": "The hostname is maccabeus.hogs.tswn.us\u001b[m and the OS is VyOS 1.3.0-rc1\u001b[m"
}
META: ran handlers
META: ran handlers

PLAY RECAP ****************************************************************************************************************************************************************************************************
maccabeus.hogs.tswn.us     : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
sdwilsh commented 3 years ago

I would have expected this line to match this and remove it, but that doesn't actually seem to be working.

sdwilsh commented 3 years ago

This is maybe an encoding issue:

$ python3
Python 3.8.5 (default, Jan 27 2021, 15:41:15) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import re
>>> r = re.compile(br"\x1b\[(\d+(;\d+)*)?m")
>>> r.search("\u001b[m".encode("cp1252"))
<re.Match object; span=(0, 3), match=b'\x1b[m'>
>>> r.search("\u001b[m")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: cannot use a bytes pattern on a string-like object
>>> r.search(b"\u001b[m")
>>>
sdwilsh commented 3 years ago

I'm also able to reproduce this on 1.4-rolling-202103210217

sdwilsh commented 3 years ago

Apparently the Ubuntu distribution of ansible 2.9 included vyos.vyos, but a rather old version, so the collection that I had installed wasn't actually being used. It didn't have the regex that I expected to match this, and that's why it was not working.