SFTP issue with libssh

[ashwini-mhatre]

SUMMARY

sftp is not working with libssh on iosxr platform

ISSUE TYPE

• Bug Report

PYLISSH and LIBSSH VERSION

(py3.6.10) amhatre@ashwinis-MacBook-Pro playbooks % pip show ansible-pylibssh
Name: ansible-pylibssh
Version: 0.2.0
Summary: Python bindings for libssh client specific to Ansible use case
Home-page: https://github.com/ansible/pylibssh
Author: Ansible, Inc.
Author-email: info+github/ansible/pylibssh@ansible.com
License: LGPLv2+
Location: /Users/amhatre/ansible_venvs/py3.6.10/lib/python3.6/site-packages
Requires: 
Required-by: 

OS / ENVIRONMENT

IOSXR 6.1.3 IOSXR 7.0.2

STEPS TO REPRODUCE

- name: test iosxr with pylibssh
  hosts: iosxr
  connection: ansible.netcommon.network_cli
  gather_facts: no
  tasks:
    - name: replace config
      register: result
      cisco.iosxr.iosxr_config:
        src: 'running_cfg_iosxr.txt'
        replace: config

EXPECTED RESULTS

changed: [127.0.0.1] => {
    "changed": true,
    "commands": [
        "load harddisk:/ansible_config.txt"
    ],
    "diff": {
        "prepared": "Building configuration...\n!! IOS XR Configuration 7.0.2\n+  interface preconfigure GigabitEthernet0/0/0/0\n   !\n+  interface preconfigure GigabitEthernet0/0/0/1\n   !\n+  interface preconfigure GigabitEthernet0/0/0/2\n   !\nend"
    },
    "invocation": {
        "module_args": {
            "admin": false,
            "after": null,
            "backup": false,
            "backup_options": null,
            "before": null,
            "comment": "configured by iosxr_config",
            "config": null,
            "exclusive": false,
            "force": false,
            "label": null,
            "lines": null,
            "match": "line",
            "parents": null,
            "provider": null,
            "replace": "config",
            "src": "hostname test1\nbanner motd \"hell\"\ntelnet vrf default ipv4 server max-servers 10\nusername vagrant\n group root-lr\n group cisco-support\n secret 10 $6$Zf8Qze0rDkx1z...$/1ly7j2qqH5vYIpGEG2a8khtTdEEGS3NnbRv6izmWCLteCQSFt2YbBCVQOfETj4A0IPquaPj8yWEqH1vXck06/\n!\nusername amhatre\n group root-lr\n group cisco-support\n!\ntpa\n vrf default\n  address-family ipv4\n   update-source dataports MgmtEth0/RP0/CPU0/0\n  !\n !\n!\ncall-home\n service active\n contact smart-licensing\n profile CiscoTAC-1\n  active\n  destination transport-method http\n !\n!\nnetconf-yang agent\n ssh\n!\ninterface Loopback888\n!\ninterface Loopback999\n!\ninterface MgmtEth0/RP0/CPU0/0\n ipv4 address dhcp\n!\ninterface preconfigure GigabitEthernet0/0/0/0\n!\ninterface preconfigure GigabitEthernet0/0/0/1\n!\ninterface preconfigure GigabitEthernet0/0/0/2\n!\nprefix-set ebpg_filter\n  192.168.0.0/16 ge 15 le 30\nend-set\n!\nprefix-set ebpg_filter2\n  192.168.0.0/16 ge 17 le 30\nend-set\n!         \nrouter static\n address-family ipv4 unicast\n  0.0.0.0/0 MgmtEth0/RP0/CPU0/0 10.0.2.2\n !        \n!         \nrouter bgp 1\n!         \nssh server v2\nssh server vrf default\nssh server netconf vrf default\nend\n"
        }
    }
}
META: ran handlers
META: ran handlers

PLAY RECAP ***********************************************************************************************************************************************************************
127.0.0.1                  : ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

ACTUAL RESULTS

<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /Users/amhatre/.ansible/tmp/ansible-local-34893sr5i_bwc/ansible-tmp-1622724615.099243-34897-263822561108764/ /Users/amhatre/.ansible/tmp/ansible-local-34893sr5i_bwc/ansible-tmp-1622724615.099243-34897-263822561108764/AnsiballZ_iosxr_config.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/Users/amhatre/ansible_venvs/py3.6.10/bin/python /Users/amhatre/.ansible/tmp/ansible-local-34893sr5i_bwc/ansible-tmp-1622724615.099243-34897-263822561108764/AnsiballZ_iosxr_config.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /Users/amhatre/.ansible/tmp/ansible-local-34893sr5i_bwc/ansible-tmp-1622724615.099243-34897-263822561108764/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
  File "/var/folders/j6/fqn4v5252b9_4ck6xxq4_rqm0000gn/T/ansible_cisco.iosxr.iosxr_config_payload_sjx_ttxa/ansible_cisco.iosxr.iosxr_config_payload.zip/ansible_collections/cisco/iosxr/plugins/module_utils/network/iosxr/iosxr.py", line 599, in copy_file
    conn.copy_file(source=src, destination=dst, proto=proto)
  File "/var/folders/j6/fqn4v5252b9_4ck6xxq4_rqm0000gn/T/ansible_cisco.iosxr.iosxr_config_payload_sjx_ttxa/ansible_cisco.iosxr.iosxr_config_payload.zip/ansible/module_utils/connection.py", line 195, in __rpc__
    raise ConnectionError(to_text(msg, errors='surrogate_then_replace'), code=code)
fatal: [127.0.0.1]: FAILED! => {
    "changed": false,
    "invocation": {
        "module_args": {
            "admin": false,
            "after": null,
            "backup": false,
            "backup_options": null,
            "before": null,
            "comment": "configured by iosxr_config",
            "config": null,
            "exclusive": false,
            "force": false,
            "label": null,
            "lines": null,
            "match": "line",
            "parents": null,
            "provider": null,
            "replace": "config",
            "src": "hostname test1\nbanner motd \"hell\"\ntelnet vrf default ipv4 server max-servers 10\nusername vagrant\n group root-lr\n group cisco-support\n secret 10 $6$Zf8Qze0rDkx1z...$/1ly7j2qqH5vYIpGEG2a8khtTdEEGS3NnbRv6izmWCLteCQSFt2YbBCVQOfETj4A0IPquaPj8yWEqH1vXck06/\n!\nusername amhatre\n group root-lr\n group cisco-support\n!\ntpa\n vrf default\n  address-family ipv4\n   update-source dataports MgmtEth0/RP0/CPU0/0\n  !\n !\n!\ncall-home\n service active\n contact smart-licensing\n profile CiscoTAC-1\n  active\n  destination transport-method http\n !\n!\nnetconf-yang agent\n ssh\n!\ninterface Loopback888\n!\ninterface Loopback999\n!\ninterface MgmtEth0/RP0/CPU0/0\n ipv4 address dhcp\n!\ninterface preconfigure GigabitEthernet0/0/0/0\n!\ninterface preconfigure GigabitEthernet0/0/0/1\n!\ninterface preconfigure GigabitEthernet0/0/0/2\n!\nprefix-set ebpg_filter\n  192.168.0.0/16 ge 15 le 30\nend-set\n!\nprefix-set ebpg_filter2\n  192.168.0.0/16 ge 17 le 30\nend-set\n!         \nrouter static\n address-family ipv4 unicast\n  0.0.0.0/0 MgmtEth0/RP0/CPU0/0 10.0.2.2\n !        \n!         \nrouter bgp 1\n!         \nssh server v2\nssh server vrf default\nssh server netconf vrf default\nend\n"
        }
    },
    "msg": "Writing to remote file [b'/harddisk:/ansible_config.txt'] failed"
}

PLAY RECAP ***********************************************************************************************************************************************************************
127.0.0.1  

### Tasks

[webknjaz]

@ashwini-mhatre please explain what doesn't work.

[ashwini-mhatre]

I am facing issue with iosxr_config Module. I am using iosxr_config to replace device configuration with source file. Internally iosxr_config module is using put method of netcommon collection and Netcommon internally using pylibssh to copy file from local system to network device. In this case I am facing "Writing to remote file [b'/harddisk:/ansible_config.txt'] failed" issue. Please find below code from where actually error is thrown https://github.com/ansible-collections/ansible.netcommon/blob/40bf3abb69c7148fbbc3ab6da7e802a76b5ea28f/plugins/connection/libssh.py#L479

[ashwini-mhatre]

@webknjaz i build pylibssh from https://github.com/ansible/pylibssh/pull/216 and following is the detailed exception which i got."msg": "Writing to remote file [b'/harddisk:/ansible_config.txt'] failed with error [Generic failure: Invalid SSH_FXP_STATUS message]"

[swietlas]

Hi, is there any update? I faced exactly the same problem. When trying to push config using iosxr_config module I get:

I'm using Ansible:

 ansible --version
ansible [core 2.13.6]
  config file = /home/swt/Repo/Automation/Ansible_demo/ACLs/ansible.cfg
  configured module search path = ['/home/swt/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/swt/.local/lib/python3.8/site-packages/ansible
  ansible collection location = /home/swt/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/swt/.local/bin/ansible
  python version = 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0]
  jinja version = 3.1.2
  libyaml = True

Collections

# /home/swt/.ansible/collections/ansible_collections
Collection        Version
----------------- -------
ansible.netcommon 3.0.1  
ansible.utils     2.6.1  
cisco.ios         4.0.0  
cisco.iosxr       4.0.2  

libssh:

pip freeze |grep libssh
ansible-pylibssh==1.0.0

Playbook: iosxr_ospf.yml

- name: Play to set up OSPF on IOSXR routers
  hosts: routersxr
  gather_facts: false
  tasks:

    - name: Collect ios_facts
      cisco.iosxr.iosxr_facts:
        gather_subset:
          - min
      register: facts
      tags: hwinfo

    - name: Generate config from jinja template
      ansible.builtin.template:
        src: iosxr_ospf.j2
        dest: ./ospfconfig/"{{ ansible_net_hostname }}".cfg
        backup: true
        mode: 0640

    - name: Setup OSPF
      cisco.iosxr.iosxr_config:
        src: ./ospfconfig/"{{ ansible_net_hostname }}".cfg
        replace: config

Jinja2 template

{% if ospf is defined %}
router ospf {{ ospf.process_id }}
router-id {{ ospf.router_id }}
 auto-cost reference-bandwidth {{ ospf.auto_cost.reference_bandwidth }}
 address-family ipv4 unicast
{% for item in ospf.areas %}
  area {{ item.area_id }}
{% for intf in item.interfaces %}
   interface {{ intf.name }}
{% if  intf.network is defined  %}
  network {{ intf.network }}
{% endif %}{#p2p#}
{% if  intf.passive is defined %}
{% if intf.passive == "true" %} 
  passive enable
{% endif %}{#-passivetrue-#}
{% endif %}{#passive#}
{% endfor %}{#intf#}
{% endfor %}{#areas#}
{% endif %}{#ospf#}

OSPF config file

router ospf 2
router-id 2.2.2.2
 auto-cost reference-bandwidth 1000
 address-family ipv4 unicast
  area 0.0.0.0
   interface GigabitEthernet0/0/0/0
  network point-to-point
  area 0.0.0.2
   interface GigabitEthernet0/0/0/1
  network point-to-point
   interface Loopback1
  network point-to-point
   passive enable

Result

TASK [Setup OSPF] **********************************************************************************************************************************************************
fatal: [R2xr]: FAILED! => {"changed": false, "msg": "Writing to remote file [b'/harddisk:/ansible_config.txt'] failed with error [Generic failure: Invalid SSH_FXP_STATUS message]"}

[blue212121]

Just an addition, when using the iosxr_user module with the public_key_contents argument (which uploads an ssh key to the device) i debugged the scp server on the iosxr side, results:

paramiko:

RP/0/RP0/CPU0:Dec 21 08:41:25.438 UTC: scp_server[67807]: main: priv:root-lr cisco-support
RP/0/RP0/CPU0:Dec 21 08:41:25.438 UTC: scp_server[67807]: main: Sink:/harddisk:/publickey_sshkeytest.b64
RP/0/RP0/CPU0:Dec 21 08:41:25.438 UTC: scp_server[67807]: In main pipe descriptors fdin=116 fdout=127 parent_out=126 parent_in=125
RP/0/RP0/CPU0:Dec 21 08:41:25.442 UTC: scp_server[67807]: communication_handler entry
RP/0/RP0/CPU0:Dec 21 08:41:25.442 UTC: scp_server[67807]: communication_handler not disconnected
RP/0/RP0/CPU0:Dec 21 08:41:25.449 UTC: scp_server[67807]: Read Length 407
RP/0/RP0/CPU0:Dec 21 08:41:25.449 UTC: scp_server[67807]: Updated Code
RP/0/RP0/CPU0:Dec 21 08:41:25.459 UTC: scp_server[67807]: Path /harddisk:/publickey_sshkeytest.b64
RP/0/RP0/CPU0:Dec 21 08:41:25.459 UTC: scp_server[67807]: Local Full /harddisk:/publickey_sshkeytest.b64
RP/0/RP0/CPU0:Dec 21 08:41:25.459 UTC: scp_server[67807]: Successful read C0664 407 publickey_sshkeytest.b64
RP/0/RP0/CPU0:Dec 21 08:41:25.459 UTC: scp_server[67807]: scpserver_download ends.

pylibssh:

RP/0/RP0/CPU0:Dec 21 08:41:45.265 UTC: scp_server[67906]: main: priv:root-lr cisco-support
RP/0/RP0/CPU0:Dec 21 08:41:45.265 UTC: scp_server[67906]: main: Sink:'/harddisk:/publickey_sshkeytest.b64'
RP/0/RP0/CPU0:Dec 21 08:41:45.265 UTC: scp_server[67906]: In main pipe descriptors fdin=130 fdout=133 parent_out=132 parent_in=131
RP/0/RP0/CPU0:Dec 21 08:41:45.488 UTC: scp_server[67906]: communication_handler entry
RP/0/RP0/CPU0:Dec 21 08:41:45.488 UTC: scp_server[67906]: communication_handler not disconnected
RP/0/RP0/CPU0:Dec 21 08:41:45.488 UTC: scp_server[67906]: %SECURITY-SSHD-3-ERR_FILE_PATH : Cannot write to system files path: ['/harddisk:/publickey_sshkeytest.b64']
RP/0/RP0/CPU0:Dec 21 08:41:45.489 UTC: scp_server[67906]: Cannot modify system files/dir
RP/0/RP0/CPU0:Dec 21 08:41:45.489 UTC: scp_server[67906]: scpserver_download ends.

to me it looks like pylibssh sends the encasing single quote which iosxr cannot handle correctly

this is confirmed with manually scp-ing the file with \'/harddisk:/publickey_sshkeytest.b64\' producing the same logs as with pylibssh

this is tested and confirmed on iosxr 6.5.1, 7.1.1, 7.5.1

[ashwini-mhatre]

@Qalthos is any update on this issue?

[hmntsharma]

Also, other than paramiko and ansible-pylibssh, is there any other library that can be used? As paramiko is allegedly very slow. Thanks!

[alexandergall]

There are really separate issues with the scp and sftp method when used with libssh in combination with some IOS-XR versions. The cisco/iosxr Ansible role uses both methods somewhat randomly, it seems.

It's libssh that adds the quotes around path names with the scp method (not pylibssh) that makes the router choke with the Cannot write to system files path diagnostic.

I switched to the sfpt method everywhere (requires patching the cisco collection since that is not configurable), but then the second problem strikes. The IOS-XR version in question violates the SFTP spec for version 3 and later that the SSH_FXP_STATUS message must include an error message and language tag. The router doesn't supply these but libssh insists they are there and raises the Invalid SSH_FXP_STATUS message error. My solution to this is to patch libssh to remove that check.

None of the problems are caused by pylibssh, I think.

[crazzy]

My solution to this is to patch libssh to remove that check.

Would you mind sharing the libssh patch as it seems to be the only way to get the iosxr_config module working currently?

[alexandergall]

Here are the patches I use (uploaded as txt since Github refuses to accept files with the .patch extension even tough "PATCH" is among the list of permtted file types). The first patch is to make the iosxr role use sftp to copy files, the second one is the libssh patch.

ansible-iosxr.txt

libssh-remove-status-msg-check.txt

[jgomezve]

@alexandergall How did you patch the libssh library ?

[Jakuje]

libssh-remove-status-msg-check.txt

Can you propose the changes to libssh in https://gitlab.com/libssh/libssh-mirror/-/merge_requests with some more information what is wrong there and how is your patch solving this issue?

[alexandergall]

libssh is correct, it's the SFTP implementation of some IOS-XR versions that's wrong. My patch is just a workaround and shouldn't be merged as it is. The reasonable thing to do would be to add a configuration option to libssh to disable the SSH_FXP_STATUS check when dealing with broken routers, I think. I don't have the time to write a complete patch right now, but I could open an issue describing the problem.

[Jakuje]

Yes please, even the issue for libssh with enough information could be enough for now just to be aware of the issue. I would rather create some quirk that would detect the broken implementation from SSH string identification or something rather than creating configuration for this bug, but that is an implementation detail that can be resolved later.

[alexandergall]

https://gitlab.com/libssh/libssh-mirror/-/issues/272