Running containerlab on `darwin`

[networkop]

Triggered by https://github.com/weaveworks/ignite/pull/860

[networkop]

I'll do a PR shortly

[networkop]

👋 @sc68cal, before I go ahead with this, there are more places where containerlab depends on Linux. For example,

$  GOOS=darwin go build -o /home/mk/clab-networkop/bin/containerlab main.go
go build github.com/containernetworking/plugins/pkg/ns: build constraints exclude all Go files in /home/mk/go/pkg/mod/github.com/containernetworking/plugins@v0.9.1/pkg/ns
$ go mod why github.com/containernetworking/plugins/pkg/ns
# github.com/containernetworking/plugins/pkg/ns
github.com/srl-labs/containerlab/clab
github.com/containernetworking/plugins/pkg/ns

So even if I disable ignite for non-Linux OS, there are still more places where this may need to be addressed. Will you be able to address these other issues?

@hellt have you had anyone trying to run containerlab on Darnwin? Do you know of any other issues that may creep up? My understanding was that MacOS runs docker in a docker-machine VM which may prevent containerlab from accessing and manipulating container network namespace.

[hellt]

@networkop containerlab won't be able to run on Darwin. the docker-machine is not providing the necessary networking primitives that containerlab uses to create the wiring plumbing

The best chance on mac so far is to use a VM with proper linux OS

[sc68cal]

@networkop containerlab won't be able to run on Darwin. the docker-machine is not providing the necessary networking primitives that containerlab uses to create the wiring plumbing

The best chance on mac so far is to use a VM with proper linux OS

Which is what I ended up doing, and mounting the git repo as a shared folder via vagrant

It's just annoying that for simple stuff where I wasn't executing the tests that were relying on the CNI, for the PR I did, I had to dig down into this super deep. Then again, this might be a golang issue compared to my normal language of Python where everything is lazily evaluated. I wrote lots of linux/iptables code for OpenStack neutron on Mac and was able to run unit tests (with the real parts mock'd out) and not being able to do something similar for this repo is annoying.

[sc68cal]

I will note that I tried to stub out other dependencies, like vishvananda/netlink#669 where there was infrastructure for doing so, that got me further along in compiling containerlab for Darwin

[networkop]

I'm no OSX expert but it looks like "developing locally and running inside a container" is a common pattern, see this.

[networkop]

That being said, I think it IS possible to run containerlab on OSX without spinning up a dedicated Linux VM. The main idea is to run containerlab from another container as opposed to from OSX's native shell. So you'd need to

1. Build a long-running docker image and place containerlab binary in it.
2. Based off this image fire up a container that would: a. Bind mount /run/netns b. Be privileged c. Run in host network mode.
3. Instead of doing containerlab <verb> run docker exec my-container containerlab <verb>

I obv. haven't tested this but, in theory, this idea should work.

[networkop]

you can take it a step further and even develop containerlab inside that docker container, e.g. if you're using VSCode this is how you'd do it

[hellt]

@networkop what about docker daemon inside this container? We will need dind I suppose

[networkop]

oh yeah, but you just need to mount the socket file. some something like -v /var/run/docker.sock:/var/run/docker/sock would do

[hellt]

That is where things will break down I'm afraid If we mount the docker socket, then the clab docker network will be attempted to be created in the docker machine I remember I was trying something like that and it was full of wonders, so I left it

Maybe someone else will be more persuasive

On Tue, 10 Aug 2021 at 12:27, Michael Kashin @.***> wrote:

oh yeah, but you just need to mount the socket file too. some something like -v /var/run/docker.sock:/var/run/docker/sock would do

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/srl-labs/containerlab/issues/577#issuecomment-895915238, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABLKV5KXYL57OGDBD4UVR2LT4D5KZANCNFSM5B2OIPKQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

[networkop]

Not if you run it in host network namespace and mount the /run/netns. There may be some other directories that need to be mounted but I believe this should be possible. After all this is how k8s CNI plugins like Cilium work - they are distributed as containers and mount the required directories from the host and from then on they manage all networking aspects for other containers on the host. @sc68cal do you want to investigate and test this?

[hellt]

@networkop there is no /run/netns on macos, so I am trying to see how this could possibly work? At the very least, it seems the netns go libs we use will have to be interfaced with darwin methods, but that is a lot of work it seems for a purpose of supporting darwin

[networkop]

there should be /run/netns inside the docker-machine. based on my understanding docker on MAC only mounts some OSX folders into this VM -- see this, so mounting anything else would mount it directly from the docker-machine VM.

[hellt]

@networkop I tried this way, but it seems that any bind mount that is defined for a lab is intercepted by docker-mac, and thus causes issues

root@docker-desktop:/# clab dep -t srlfrr01.clab.yml
INFO[0000] Parsing & checking topology file: srlfrr01.clab.yml
INFO[0000] Pulling docker.io/frrouting/frr:v7.5.0 Docker image
INFO[0007] Done pulling docker.io/frrouting/frr:v7.5.0
WARN[0007] it appears that container host has low memory available: ~0Gi. This might lead to runtime errors. Consider freeing up more memory.
INFO[0007] Creating lab directory: //clab-srlfrr01
INFO[0007] Creating root CA
INFO[0007] Creating docker network: Name='clab', IPv4Subnet='172.20.20.0/24', IPv6Subnet='2001:172:20:20::/64', MTU='1500'
INFO[0008] Creating container: frr
INFO[0008] Creating container: srl
ERRO[0008] failed deploy phase for node "srl": Error response from daemon: Mounts denied:
The path /clab-srlfrr01/srl/config.json is not shared from the host and is not known to Docker.
You can configure shared paths from Docker -> Preferences... -> Resources -> File Sharing.
See https://docs.docker.com/docker-for-mac for more info.
ERRO[0008] failed deploy phase for node "frr": Error response from daemon: Mounts denied:
The path /daemons is not shared from the host and is not known to Docker.
You can configure shared paths from Docker -> Preferences... -> Resources -> File Sharing.
See https://docs.docker.com/docker-for-mac for more info.
INFO[0008] Creating virtual wire: srl:e1-1 <--> frr:eth1
ERRO[0008] failed to Statfs "": no such file or directory
ERRO[0008] failed to run postdeploy task for node frr: failed to Statfs "": no such file or directory
INFO[0008] Adding containerlab host entries to /etc/hosts file
+---+-------------------+--------------+-----------------------+-------+-------+---------+--------------+--------------+
| # |       Name        | Container ID |         Image         | Kind  | Group |  State  | IPv4 Address | IPv6 Address |
+---+-------------------+--------------+-----------------------+-------+-------+---------+--------------+--------------+
| 1 | clab-srlfrr01-frr | 6a0e94f2af6c | frrouting/frr:v7.5.0  | linux |       | created | NA           | NA           |
| 2 | clab-srlfrr01-srl | f726ee6ecee5 | ghcr.io/nokia/srlinux | srl   |       | created | NA           | NA           |
+---+-------------------+--------------+-----------------------+-------+-------+---------+--------------+--------------+

[networkop]

Does the networking side of it work if you don't have any bind mounts though? This is why I said that you may need to mount a few extra directories or specify which paths you can mount from, e.g. if /Users is mounted inside the docker-machine on the same path, you should be able to bind mound this into a containerlab container as /Users:/Users and get that visibility.

[hellt]

@networkop thanks, that got me further

if I run a container on a mac like that

docker run --rm -it --network host --privileged \
       -v /var/run/docker.sock:/var/run/docker.sock \
       -v /run/netns:/run/netns \
       -v /proc:/proc \
       -v /Users/romandodin/clab:/Users/romandodin/clab \
       ubuntu bash

after launching this container I installed the docker-cli and containerlab with apt update && apt -y install curl && curl -L https://gist.githubusercontent.com/hellt/f5de958af290f9a239dec07da4231766/raw/3d7c3ac2878ff25ebe641402a882e0d412f5dd81/clab.sh | bash

where the last mount is the dir under /Users that exists on mac, then you can install docker-cli and containerlab there and that almost work

then it almost works

root@docker-desktop:/Users/romandodin/clab# clab dep -t srlfrr01.clab.yml
INFO[0000] Parsing & checking topology file: srlfrr01.clab.yml
INFO[0000] Creating lab directory: /Users/romandodin/clab/clab-srlfrr01
INFO[0000] Creating root CA
INFO[0000] Creating docker network: Name='clab', IPv4Subnet='172.20.20.0/24', IPv6Subnet='2001:172:20:20::/64', MTU='1500'
INFO[0000] Creating container: srl
INFO[0001] Creating container: srl2
INFO[0002] Creating virtual wire: srl:e1-1 <--> srl2:e1-1
ERRO[0002] Failed to open current namespace: failed to Statfs "/proc/3900/task/3907/ns/net": no such file or directory
INFO[0002] Adding containerlab host entries to /etc/hosts file
INFO[0002] 🎉 New containerlab version 0.16.2 is available! Release notes: https://containerlab.srlinux.dev/rn/0.16/#0162
Run 'containerlab version upgrade' to upgrade or go check other installation options at https://containerlab.srlinux.dev/install/
+---+--------------------+--------------+-----------------------+------+-------+---------+----------------+----------------------+
| # |        Name        | Container ID |         Image         | Kind | Group |  State  |  IPv4 Address  |     IPv6 Address     |
+---+--------------------+--------------+-----------------------+------+-------+---------+----------------+----------------------+
| 1 | clab-srlfrr01-srl  | b6d7a5aab012 | ghcr.io/nokia/srlinux | srl  |       | running | 172.20.20.2/24 | 2001:172:20:20::2/64 |
| 2 | clab-srlfrr01-srl2 | feed9d422da8 | ghcr.io/nokia/srlinux | srl  |       | running | 172.20.20.3/24 | 2001:172:20:20::3/64 |
+---+--------------------+--------------+-----------------------+------+-------+---------+----------------+----------------------+

there is some issue with root NS detection it seems, I think it comes from this step https://github.com/srl-labs/containerlab/blob/3087f30da9017f47ab68ac0e22a7cb94c8ddc8ff/clab/netlink.go#L179

and I am not sure why it fails...

[networkop]

ah, ok you need to see the process ids of the host, I'm not 100% what's the right docker command for it. can you try --cgroupns 'host' ?

[networkop]

in k8s it's pod.Spec.hostPID = true so it must be possible to do the same with docker

[networkop]

ah, there it is , i think --pid="host"

[hellt]

@networkop looks like this did it I am testing it atm, but looks promising so far

[hellt]

@networkop you are A W E S O M E, man

root@docker-desktop:/Users/romandodin/clab# clab dep -t srlfrr01.clab.yml
INFO[0000] Parsing & checking topology file: srlfrr01.clab.yml
INFO[0000] Creating lab directory: /Users/romandodin/clab/clab-srlfrr01
INFO[0000] Creating docker network: Name='clab', IPv4Subnet='172.20.20.0/24', IPv6Subnet='2001:172:20:20::/64', MTU='1500'
INFO[0000] Creating container: srl2
INFO[0000] Creating container: srl
INFO[0001] Creating virtual wire: srl:e1-1 <--> srl2:e1-1
INFO[0001] Adding containerlab host entries to /etc/hosts file
+---+--------------------+--------------+-----------------------+------+-------+---------+----------------+----------------------+
| # |        Name        | Container ID |         Image         | Kind | Group |  State  |  IPv4 Address  |     IPv6 Address     |
+---+--------------------+--------------+-----------------------+------+-------+---------+----------------+----------------------+
| 1 | clab-srlfrr01-srl  | 3513a7f4aed6 | ghcr.io/nokia/srlinux | srl  |       | running | 172.20.20.2/24 | 2001:172:20:20::2/64 |
| 2 | clab-srlfrr01-srl2 | 191de529f549 | ghcr.io/nokia/srlinux | srl  |       | running | 172.20.20.3/24 | 2001:172:20:20::3/64 |
+---+--------------------+--------------+-----------------------+------+-------+---------+----------------+----------------------+
root@docker-desktop:/Users/romandodin/clab# docker exec  clab-srlfrr01-srl ip l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
5: gway-2800@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether fe:70:87:f0:23:be brd ff:ff:ff:ff:ff:ff link-netns srbase-mgmt
6: monit_in@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9234 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 82:28:01:55:8b:f0 brd ff:ff:ff:ff:ff:ff link-netns monit
7: mgmt0-0@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether a6:08:d2:99:bc:a5 brd ff:ff:ff:ff:ff:ff link-netns srbase-mgmt
    alias mgmt0.0
8: dummy-mgmt0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 36:a7:c8:bd:44:84 brd ff:ff:ff:ff:ff:ff
75: mgmt0@if76: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1514 qdisc noqueue state UP mode DEFAULT group default
    link/ether 02:42:ac:14:14:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
80: e1-1@if79: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9500 qdisc noqueue state UP mode DEFAULT group default
    link/ether aa:c1:ab:7d:cf:75 brd ff:ff:ff:ff:ff:ff link-netnsid 1

--{ running }--[  ]--
A:srl2# ping 192.168.1.1 network-instance default
Using network instance default
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=35.0 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=4.90 ms
^C
--- 192.168.1.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 4.900/19.971/35.042/15.071 ms

[networkop]

nice 😀 , all these hours of me helping our mac developers at my previous company have not gone to waste. I think the same approach can be applied to docker on windows, although I, personally, use docker inside a WSL VM, without any desktop integration.

[hellt]

@networkop definitely I will try to come up with a diagram how this works (in September), unless anyone else can do that.

For now I will create a container image with the tools/containerlab installed, so others could use it already Next step would be to incorporate container build with goreleaser for each release

[hellt]

this has been sorted out and documented here https://containerlab.srlinux.dev/install/#mac-os

@networkop feel free to close this, if that answers the initial reqs

[networkop]

I think this is as far as we can get on OSX. Development on OSX can be done in a similar way - from inside a container with VSCode plugins. @sc68cal feel free to re-open if you think more can be done here.

[sc68cal]

Thanks, appreciate everyone's effort on this.