search

ipspace / netlab

Making virtual networking labs suck less
https://netlab.tools
Other
428 stars 63 forks source link

Adding VyOS support in devcontainer #1261

Closed ubaumann closed 1 month ago

ubaumann commented 1 month ago

I am using the netlab devcontainer with Codespaces and when I start a topology with VyOS, I get the following error:

provider clab: executing sudo -E containerlab deploy -t clab.yml
INFO[0000] Containerlab v0.55.0 started                 
INFO[0000] Parsing & checking topology file: clab.yml   
Error: failed to verify bind path: stat /lib/modules: no such file or directory
Error executing sudo -E containerlab deploy -t clab.yml:
  Command '['sudo', '-E', 'containerlab', 'deploy', '-t', 'clab.yml']' returned non-zero exit status 1.
[FATAL]   netlab up: sudo -E containerlab deploy -t clab.yml failed, aborting...

To be able to start the container I needed to install linux-headers-generic:

$ sudo apt install linux-headers-generic
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Note, selecting 'linux-headers-amd64' instead of 'linux-headers-generic'
The following additional packages will be installed:
  linux-compiler-gcc-12-x86 linux-headers-6.1.0-23-amd64 linux-headers-6.1.0-23-common linux-kbuild-6.1 linux-libc-dev
The following NEW packages will be installed:
  linux-compiler-gcc-12-x86 linux-headers-6.1.0-23-amd64 linux-headers-6.1.0-23-common linux-headers-amd64 linux-kbuild-6.1
The following packages will be upgraded:
  linux-libc-dev
1 upgraded, 5 newly installed, 0 to remove and 65 not upgraded.
Need to get 15.4 MB of archives.
After this operation, 64.6 MB of additional disk space will be used.
Do you want to continue? [Y/n]

When inspecting the container logs I see the following errors:

[FAILED] Failed to start auditd.ser…e - Security Auditing Service.
See 'systemctl status auditd.service' for details.
[ 8640.607657] vyos-router[452]: Traceback (most recent call last):
[ 8640.607918] vyos-router[452]:   File "/usr/libexec/vyos/conf_mode/system_option.py", line 190, in <module>
[ 8640.608247] vyos-router[452]:     generate(c)
[ 8640.608532] vyos-router[452]:   File "/usr/libexec/vyos/conf_mode/system_option.py", line 108, in generate
[ 8640.608794] vyos-router[452]:     grub_util.update_kernel_cmdline_options(' '.join(cmdline_options))
[ 8640.609073] vyos-router[452]:   File "/usr/lib/python3/dist-packages/vyos/system/image.py", line 275, in wrapper
[ 8640.609332] vyos-router[452]:     ret = func(*args, **kwargs)
[ 8640.609542] vyos-router[452]:           ^^^^^^^^^^^^^^^^^^^^^
[ 8640.609677] vyos-router[452]:   File "/usr/lib/python3/dist-packages/vyos/system/grub_util.py", line 70, in update_kernel_cmdline_options
[ 8640.609783] vyos-router[452]:     set_kernel_cmdline_options(cmdline_options, version, root_dir)
[ 8640.609892] vyos-router[452]:   File "/usr/lib/python3/dist-packages/vyos/system/compat.py", line 335, in wrapper
[ 8640.610006] vyos-router[452]:     render_grub_cfg()
[ 8640.610111] vyos-router[452]:   File "/usr/lib/python3/dist-packages/vyos/system/compat.py", line 325, in render_grub_cfg
[ 8640.610351] vyos-router[452]:     fields = grub_cfg_fields(root_dir)
[ 8640.610549] vyos-router[452]:              ^^^^^^^^^^^^^^^^^^^^^^^^^
[ 8640.610886] vyos-router[452]:   File "/usr/lib/python3/dist-packages/vyos/system/compat.py", line 301, in grub_cfg_fields
[ 8640.611279] vyos-router[452]:     menu_entries = update_version_list(root_dir)
[ 8640.612601] vyos-router[452]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ 8640.616111] vyos-router[452]:   File "/usr/lib/python3/dist-packages/vyos/system/compat.py", line 247, in update_version_list
[ 8640.616329] vyos-router[452]:     menu_entries = parse_menuentries(grub_cfg_main)
[ 8640.616583] vyos-router[452]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ 8640.616814] vyos-router[452]:   File "/usr/lib/python3/dist-packages/vyos/system/compat.py", line 152, in parse_menuentries
[ 8640.617005] vyos-router[452]:     config_text = Path(grub_path).read_text()
[ 8640.617194] vyos-router[452]:                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ 8640.617490] vyos-router[452]:   File "/usr/lib/python3.11/pathlib.py", line 1059, in read_text
[ 8640.617675] vyos-router[452]:     with self.open(mode='r', encoding=encoding, errors=errors) as f:
[ 8640.617857] vyos-router[452]:          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ 8640.618057] vyos-router[452]:   File "/usr/lib/python3.11/pathlib.py", line 1045, in open
[ 8640.618250] vyos-router[452]:     return io.open(self, mode, buffering, encoding, errors, newline)
[ 8640.618433] vyos-router[452]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ 8640.618671] vyos-router[452]: FileNotFoundError: [Errno 2] No such file or directory: '//boot/grub/grub.cfg'
[ 8640.686466] vyos-router[279]: could not reset system option files ... failed!
[ 8641.021044] vyos-router[459]: sysctl: cannot stat /proc/sys/net/ipv4/neigh/default/gc_thresh3: No such file or directory

I assume I can ignore the errors for now. The router seems to work. At least OSPF and BGP is up and the ping works.

Can we add linux-headers-generic to the devcontainer for additional support? It increases the image size by approximately 64.6 MB.

Regards, Urs

ssasso commented 1 month ago

Hi @ubaumann, unfortunately it seems it's not only the linux headers/modules missing, but the procfs is not mounted/not correctly exposing all the parameters required for vyos to work.

IIRC, it's also not possible to load additional kernel modules from a devcontainer, and vyos requires multiple of them (that's why the /lib/modules mountpoint for vyos container).

can you please try a modprobe vrf or lsmod from the devcontainer?

(side note: it's not enough to add to the container the modules related to the kernel shipped with the container distro image, vyos requires the modules of the kernel running on the host)

ssasso commented 1 month ago

BTW, I see no issue in adding the package you spotted, but we also need to write somewhere in the documentation (device caveats?) what are the limitations of VyOS on a devcontainer.

If you have a device running it could be great if you can do some more testing and see what's working and what is not

ipspace commented 1 month ago

Hi @ubaumann, like @ssasso I see no problem adding that package, but we have to make sure it solves the problem in the target environment (= GitHub Codespaces). Unfortunately, you cannot load additional modules within Codespaces Ubuntu instance, so if a container depends on either having a kernel module like VXLAN or being able to load one, it will probably fail.

However, if VyOS works for you in Codespaces after that fix, let's do it (plus documenting the limitations).