scrapli-netconf 2021.7.30 - no requested output returned from device

[dmulyalin]

Hi, using Scrapli-Netconf with containerized Arista EOS image running docker on CentOS7:

[root@~]# hostnamectl
    Virtualization: vmware
  Operating System: CentOS Linux 7 (Core)
       CPE OS Name: cpe:/o:centos:centos:7
            Kernel: Linux 3.10.0-862.el7.x86_64
      Architecture: x86-64

[root@~]# docker exec -it ceos1 Cli
ceos1>show version
Software image version: 4.26.0F-21792469.4260F (engineering build)

ceos1#show run ?
management api netconf
   transport ssh def

Everything works fine using scrapli-netconf 2021.1.30:

[root@test]# python3 -m pip list | grep scrapli
nornir-scrapli        2021.7.30
scrapli               2021.7.30
scrapli-cfg           2021.7.30
scrapli-community     2021.7.30
scrapli-netconf       2021.1.30
[root@test]# 

[root@test]#  cat test.py 
from scrapli_netconf.driver import NetconfDriver

my_device = {
    "host": "10.0.1.4",
    "auth_username": "nornir",
    "auth_password": "nornir",
    "auth_strict_key": False,
    "port": 830
}

conn = NetconfDriver(**my_device)
conn.open()
response = conn.get_config(source="running")

print(response.result)

[root@test]# python3 test.py 
/usr/local/lib/python3.6/site-packages/scrapli/decorators.py:455: FutureWarning: `comms_ansi` argument will be deprecated at the 2022.01.30 release (and any pre-releases),please modify your code to no longer pass this argument. The stripping of ANSI characters will now happen automatically if an ANSI escape is seen in device output.
  warn(self.warning, FutureWarning)
<rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="101">
  <data time-modified="2021-09-01T09:27:43.379730871Z">
    <acl xmlns="http://openconfig.net/yang/acl"/>
    <arista xmlns="http://arista.com/yang/experimental/eos">
      <eos>
        <bridging xmlns="http://arista.com/yang/experimental/igmpsnooping">
          <igmpsnooping>
            <config/>
          </igmpsnooping>
        </bridging>
        <mlag xmlns="urn:aristanetworks:yang:experimental:eos">
          <config>
            <dual-primary-action>action-none</dual-primary-action>
            <enabled>true</enabled>

... lots of output proceeds ...

But same code stops working with scrapli-netconf 2021.7.30 - no configuration returned from device:

[root@test]# python3 -m pip install scrapli_netconf --upgrade
...
Installing collected packages: scrapli-netconf
  Attempting uninstall: scrapli-netconf
    Found existing installation: scrapli-netconf 2021.1.30
    Uninstalling scrapli-netconf-2021.1.30:
      Successfully uninstalled scrapli-netconf-2021.1.30
Successfully installed scrapli-netconf-2021.7.30

[root@test]# python3 -m pip list | grep scrapli
nornir-scrapli        2021.7.30
scrapli               2021.7.30
scrapli-cfg           2021.7.30
scrapli-community     2021.7.30
scrapli-netconf       2021.7.30
[root@test]# 

[root@test]# python3 test.py 
<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="101">
  <get-config>
    <source>
      <running/>
    </source>
  </get-config>
</rpc>

[root@test]# 

Am I missing some extra settings for NetconfDriver or doing something wrong?

[carlmontanari]

Hey @dmulyalin

Thanks for opening this! Nope, don't think you are doing anything wrong!

Can't say I'd tested this with EOS before -- it looks like the addition of -tt to force a pty when spawning a netconf session is to blame. This flag was added to support sending huge commands basically but has been a pain... 🥲 without this or mucking w/ stty icanon settings we would hit max line lengths of 1024 and fall over (see #46).

Good news -- should be an easy fix. Bad news -- its not the most elegant thing in the world. The fix should be adding the following argument to your driver creation: "transport_options": {"netconf_force_pty": False}. The netconf_force_pty -> false disables the -tt flag that was added more recently. Until this issue I had only seen this be an issue w/ some junos boxes that decide they dont want to allocate a pty on port 830 for some reason.

If you do use this flag and you want to send a giant payload, I suspect you'll hit the line/char limit thing -- in which case you can set use_compressed_parser -> False. Theres another whole backstory to that in an issue somewhere but TLDR some devices (juniper) started sending rpc replies at the end of the rpc-request message but before the newline and closing ## anchor -- so since there is no point of sending "pretty" xml and to work around this the parser removes newlines which causes us to send a giant single line of xml.

Alternatively you could use ssh2/paramiko/asyncssh transports which should not have the same challenges.

I've really wanted to not have any device specific "profiles"/"drivers" in scrapli netconf but it seems more and more like it may be a good idea to just have sane defaults for devices since there seems to be enough little annoying tweaks that need to be made (esp in context of system transport due to all the extra stuff in place to make that work).

In summary.... netconf is a pita sometimes 😁 and let us know if netconf_force_pty helps out and/or you give the other transports a shot!

Carl

PS - doubt that it should matter, but was recreating/testing this w/ 4.26.2F ceos.

[dmulyalin]

Hi @carlmontanari, thank you for prompt response.

Tried both "transport_options": {"netconf_force_pty": False} and transport: paramiko - no luck. Calling get_config multiple times sometimes return configuration, sometimes echoes back rpc content. Have two cEOS instances running, this behaviour is the same for both of them.

[carlmontanari]

Thanks for the update!

I'll try to get a deeper dive into this this coming weekend and will keep ya posted!

Carl

[carlmontanari]

Sorry for the slow response -- got a little side-tracked!

Started to take a look at this this morning and realized that I had thought I mentioned (but clearly did not -- my fault!!) that the netconf_force_pty flag will require installing the pre-release version of scrpali-netconf. This dict is basically ignored by the transports unless there is an option configured, so it just kinda silently passed before.

This pre-release version should be handily pip installable pip install scrapli-netconf==2022.01.30a1.

With that and the netconf_force_pty flag I'm unable to reproduce. If I revert back to 2021.07.30 I can very much reproduce the error and even break it in other ways it looks like 😁. So I think/hope that will sort us out!

Again, sorry for the slow response, and thanks again for opening this and for the great detail!

Carl

[carlmontanari]

Gunna close this for now, but please defo re-open if my last comments were not accurate/there is something broken here! Appreciate you opening this and the support!!

Carl

[dmulyalin]

@carlmontanari updating to scrapli-netconf==2022.01.30a1 coupled with netconf_force_pty=False solves this problem, thank you for looking at it.

[carlmontanari]

Awesome! Thanks so much for confirming!