Replace telnetlib with scrapli

[kaelemc]

This PR replaces the usage of telnetlib in vrnetlab with scrapli.

• Closes #228 and #164

Changes

• Telnetlib has been removed and replaced and now the Scrapli Base Driver class is used for base telnet functionality.

  • Previously telnetlib was used to connect to the qemu monitor, I didn't see any use for this anywhere so I removed this functionality as well
  • All nodes have had old telnetlib functions swapped with new scrapli-based ones.
  • All Dockerfiles except OpenWRT (and maybe one or two more i'm forgetting) should be on debian:bookworm-slim from the Amazon ECR.
  • Dockerfiles have had git and pip added, as well as installation command of Scrapli via the git repo.

• Cisco and Juniper nodes are in their own vendor-specific subdirectory.

  • Defined a quick and dirty VENDOR_SUBDIR variable in the Makefiles of each node as well as logic in makefile.include so that files get copied one level further (due to vendor subdirectory). If VENDOR_SUBDIR is 1 then files are copied one subdirectory level further to account for the vendor subdirectory.

• Logger for VR class (inherited by all nodes) now uses a logger under the vrnetlab instance. This is because Scrapli uses the root logging instance, and outputs all channel output as 'debug' log level.

  • The root logger is now set to only write 'info' level logs.
  • self.logger will still write 'debug' logs to stdout (docker logs).

• Coloured the logging levels so it's easier to see, looks nicer visually too.

Functions

read_until()

telnetlib had a read_until() function which was used by wait_write. This has been re-implemented under the VR class.

Args:

• match_str: a string of the text to wait for
• timeout: a float of how long to wait before exiting the function even if no match is found. Defaults to None.

Returns: All console output that was read up until the match, or function timeout.

wait_write()

Adapted to scrapli, functionality should be analogous to the telnetlib version.

• Added argument timeout (float): This has been added, and is passed to read_until, Defaults to None.

expect()

telnetlib had the expect() function which was used in the launch.py entrypoint. This new adapted version in the VR class should function the same as the telnetlib one.

Args:

• regex_list: a list of byte-strings which are used to match via regex on the console output.
• timeout: a float of how long before the function should just timeout and stop waiting. Defaults to None.

Returns: a tuple of:

• List index of the matched byte-string. Defaults to -1 if there was no match. (int)
• The regex match object. (re.match)
• The console output up until that point (byte-string)

print()

The VR class now has a print function which is used to simply write to stdout and instantly flush. It's used so that the telnet console can output nicely on docker logs.

If the console output was printed via the logger, the formatting would make it difficult to interpret the output.

Args:

• bytes: byte-string of what to write to stdout.

Usage

Relevant changes to make a node work:

• Ensure git, pip and scrapli are installed in the Dockerfile

• Replace self.tn.expect() with self.expect()

• Use self.print() to print any console output, do not use logging for this. There is no need to decode()/convert the returned byte-string of expect() as self.print() doesn't accept strings, this would result in malformed outputs.

• Logging levels for any log output should be changed to be more appropriate.

• INFO is Green

• WARNING is Yellow

• ERROR/DEBUG is Red

Other

• Added an XRv_qcow folder to build XRv images out of qcow2 images, existing XRv builds out of vmdks
• XRv9k, CSR, cat8kv and cat9kv nodes use their relevant Scrapli driver.
• XRv didn't play nicely with the XR driver, I assume due to poor performance and old code version (XRv is deprecated and only runs 6.x code, XRv9k is 7.x)
• ASAv was not working correctly, had to make fixes for that.

I don't expect other nodes to 'plug and play' and work perfectly from the get go, I'm sure I have made errors or overlooked something so some work is required on the other nodes to ensure functionality.

• For other nodes I think the Scrapli drivers can probably increase reliability and make it easier to send configs.

Collaboration from others is required so we can confirm all nodes work reliably (as well as to fix some other possible issues in the way i've implemented the changes). I'm open to feedback 😊.

Confirmed working platforms

subdirectory names are in brackets. Follows alphabetical/subfolder order of the directory tree.

• [ ] Aruba AOS-CX
• [ ] Cisco ASAv
• [x] Cisco Cat8kv (c8000v)
• [x] Cisco Cat9kv
• [x] Cisco CSR1kv (csr)
• [ ] Cisco FTDv
• [ ] Cisco Nexus 9k (n9kv)
• [ ] Cisco NX-OS
• [ ] Cisco IOSv (vios)
• [x] Cisco XRv (.vmdk)
• [x] Cisco XRv (.qcow2) (xrv_qcow)
• [x] Cisco XRv9k
• [ ] cmglinux
• [ ] Dell SONiC
• [ ] Fortigate
• [ ] FreeBSD
• [ ] Dell OS10 (ftosv)
• [ ] Huawei VRP
• [ ] Juniper vJunosEvolved
• [ ] Juniper vJunosRouter
• [ ] Juniper vJunosSwitch
• [ ] Juniper vMX
• [ ] Juniper vQFX
• [ ] Juniper vSRX
• [ ] IPInfusion OcNOS
• [ ] OpenBSD
• [ ] OpenWRT
• [ ] Palo Alto PA-VM (pan)
• [ ] MikroTik ROS (routeros)
• [ ] SONiC
• [ ] Nokia SROS
• [ ] Ubuntu
• [ ] Arista vEOS
• [ ] Huawei VRP (again?) (vrp)
• [ ] HP vsr1000

[kaelemc]

It might be worth making another scrapli branch and then I can change this PR to merge into that branch (instead of master).

This way other contributors can submit PRs for any changes to other nodes and once all/enough nodes work, that branch can be merged into master?

[hellt]

sound idea @kaelemc I have created the scrapli-dev branch

[kaelemc]

IOS-XE nodes (CSR1000V, Cat8kv and Cat9kv) the Scrapli XE driver is now being used. Had some issues with it working with IOS-XE 16.x but it was an error on my end.. got the configs (both bootstrap and startup) to work nicely.

[kaelemc]

Maybe this belongs as it's own PR but I've added an xrv_qemu directory. It is the same as XRv but with some modifications to make XRv work when the user provides the qcow2 image.

The existing xrv directory requires vmdk images, most users who import images from CML or other places on the web most likely will get a qcow2, this makes it easier for users so they don't have to mess around with qemu-img. (even the vmdk I have has been converted from a qcow via qemu-img)

I tried one more time to use the scrapli IOSXRDriver with XRv but it still seems unreliable, and I get some weird behavior from the XRv (don't think this is scrapli's fault).

[jbemmel]

Not a trivial change, but I would suggest to create 1 base Dockerfile for all of vrnetlab, and then derive all platform images from that

It doesn't make sense to do 100x "install scrapli" in all those separate files - that becomes unmanageable

[hellt]

yeah we can definitely create the base image hosted on ghcr.io with base pacakages that every vr system relies on

[kaelemc]

@jbemmel Excellent idea, certainly is/was unmanageable for me

[kaelemc]

So this is a difficult one to balance, in labs of different sizes the nodes take varying times to boot (depending on system specs etc.).. I'm wondering if we should bump all the scrapli timeouts to something very large to make the connection timing out a non-issue?

Currently I have XE devices on 10 minutes, I have a feeling for people with lower-spec systems might hit time-outs meaning they could never boot the device. Should we increase to something very large just to be safe?

The base Scrapli Driver (underlying telnet connection for wait_write, expect and read_until functions) uses a timeout of 1hr.

[hellt]

I agree, let's have a long enough timeout so that you don't have to guess the time down to minutes.

We should leave the timeout configurable via the env var setting that can be then put in the topology file if needed to tune it up

[kaelemc]

Ok 👍, having timeouts as an env var is a good idea

[kaelemc]

I had to remove NETCONF configuration on CSR as different versions of IOS-XE 16.x and 17.x have different behaviors when this command is entered in the config.

In 16.x a prompt will appear asking for some confirmation about the NETCONF configuration. I figured it's best to just remove this instead of adding extra complexity to handle the prompt for that single command.

Thoughts?

[kaelemc]

SROS seems to be working fine with minimal changes.

I just had to comment out the defaulting of the clean_buffer=true in wait_write() (since this is not supported anymore). If I run into issues I'll implement the functionality back in.

Also removed the env var printing from the SROS side, This is done as part of vrnetlab now.

[kaelemc]

One other thing I noticed and haven't really been able to crack (granted I haven't invested too much time into it).

With telnetlib it seemed like we got the raw bytes out of the console. When booting images you'd see the menus and console outputs as if you were directly consoled into the node. There were other things like the console outputs being coloured on some nodes which did that.

This is not the case with Scrapli. The boot menu is not displayed like it was with telnetlib and on nodes like XRv9k we've lost the coloured output.

I don't think this is a case of ANSI codes not being recognised by the logger or my terminal. Not sure exactly what is going on with that.

This seems to be purely a cosmetic thing. In my view it is pretty much irrelevant, but thought I might just put the information out there.

[hellt]

could be that scrapli strips ansi chars by default maybe @carlmontanari knows/rememembers if there is a way to have a clear channel when doing telnet-like bits

[kaelemc]

@hellt If you have any spare time, kindly could you get a 'master' image on ghcr up and running. I think it'd be good to continue further development using the image.

I'll handle the work of updating all the Dockerfiles :).

I think you can pretty much copy what the SROS Dockerfile installs, add genisoimage since it looks like Cisco and Juniper images are using that.

Some images have OVS installed as well, although doesn't seem to be used.

[kaelemc]

Of course install pip and scrapli too! (via the git source is preferred for now, just like the current Dockerfiles) :)

[hellt]

@kaelemc please try ghcr.io/srl-labs/vrnetlab-base:0.0.1. I think it has all the deps required for all the images. The dockerfile for this one is added in 13b3cc6

[carlmontanari]

there is no way to not strip ansi using "normal" scrapli stuff, but you could easily replace the channel.read method w/ your own that does not strip if you were so inclined. but -- you will break all the "normal" scrapli behavior if there is ansi stuff (because it would cause failed matches on prompts/inputs and stuff). I dont think anyone has ever not wanted to strip it or ask about it so this is a first :P

[kaelemc]

@carlmontanari Thanks for replying, yea I guess the usage here in vrnetlab is a little unorthodox 😄 but that's fine. Personally I don't think it's too much of an issue anyway, certainly not worth it if we lose behaviour from Scrapli.

Would it be ok if you could cut a new release of Scrapli? https://github.com/carlmontanari/scrapli/pull/355 fixes an issue I was running into on a lot of platforms but the last release was from July. Ideally of course we'd want to pin the containers to a specific release :).

@hellt Thx, the base image is working great. Just if you could install the genisoimage pkg that'd be great, a few of the Cisco + Juniper platforms rely on it.

[hellt]

@kaelemc added genisoimage and reuploaded 0.0.1 image

[kaelemc]

@hellt Thanks, I guess one more thing is if you could just change Scrapli to install from the latest changes via:

pip install git+https://github.com/carlmontanari/scrapli --break-system-packages

Otherwise everything is broken and won't work.. (seems to be a random EOF sent at some point from either qemu or the node itself and Scrapli will think the connection hasn't been opened correctly)

If/when a new release comes out with the fixes then we can pin the version to that release :)

[carlmontanari]

@kaelemc 2024.7.30.post1 just pushed to pypi 🫡

[hellt]

thanks Carl @kaelemc I have reuploaded the base image with the new version

[kaelemc]

@carlmontanari @hellt Thanks!

[kaelemc]

@hellt New image (& scrapli release) are all working great, i've moved Cisco nodes over so far and tested all of them. Install time is much better without having to re-install all the packages (even better when everything is cached locally of course 😄).

I have the Dockerfile changes ready for all the other nodes but I want to test as much as I can first to see if any key packages are missing, any incompatibilities etc.

In terms of Dockerfiles, since the MAINTAINER tag is now deprecated, should I migrate those over to labels or remove them entirely, it seems some of these may have been copied over from the original vrnetlab so possibly those users aren't active, or contributing to hellt/vrnetlab? I'm also seeing inconsistencies in how the maintainer labels are handled (LABEL maintainer vs LABEL org.opencontainers.image.authors) and only a few Dockerfiles have these labels anyway.

and in the recent commits i've just pushed you'll see on XRv9k I've switched it over to pull vcpu and ram from the env vars directly instead of via the flags/argparser. I figured if we want vrnetlab to be the 'source of truth' for those things (discussed in #2285.) there's no need to have the clab end to contain logic to pass vcpu/ram via the flags... do you agree with this approach, am I clouding this PR with too much other stuff, what are your thoughts?

I've also implemented it this way with the cat8kv (for some reason there was no vcpu/ram knobs implemented).

[hellt]

you can remove the maintainer label/field altogether. It is not up to date anyways.

Yes, correct, the defaults for the cpu/mem should stay within the vrnetlab node definition, and containerlab can override this via QEMU

[kaelemc]

you can remove the maintainer label/field altogether. It is not up to date anyways.

👍

Yes, correct, the defaults for the cpu/mem should stay within the vrnetlab node definition, and containerlab can override this via QEMU

Alright, currently I have it set to use env vars called VCPU and RAM as that's what I've generally always used (it's what containerlab is using. It's also in the xrv9k docs in the 'resource requirements' admonition.

Didn't know about the QEMU_SMP/QEMU_MEMORY but I can easily changeover if you think this is the preferred naming?

[hellt]

it is not only the preferred naming, but also is generically loaded in the vrnetlab common

The reason there are VCPU/MEM dangling in the code base is pure legacy. We should converged on the global QEMU_* env vars as they are read once by every node, without duplication

[kaelemc]

Oh perfect, got it. Will make the changes 👍

[kaelemc]

Will convert as many of the nodes to use the Scrapli community drivers as possible.

I've added a checklist in the original PR comment of nodes I think are 'working' for sure.

• Unlike XRv9k, regular XRv isn't using the IOS-XR driver because it's very much legacy and doesn't want to play nice.
• All XE nodes are using IOSXEDriver and work perfectly
• In terms of Cisco, FTDv, ASAv, Nexus devices and vIOS I will try to move to their respective drivers (vIOS runs classic IOS but should work with the XE Driver)

Every node is using the new base vrnetlab image, will have to iron out any missing packages or incompatibilities there.

All Cisco nodes have default SMP/memory and I have tested that they are infact overriden correctly using the respective env vars as pointed out. I had to make a minor change for XRv9k as without the explicit single socket config it did nothing but kernel panic.

This means like other nodes, for XRv9k users can just set the QEMU_SMP to the number of vCPUs they want instead of QEMU_SMP: cores=x,threads=1,sockets=1. But if they want to supply it like this they still can.