search

metal-stack / mini-lab

a small, virtual setup to locally run the metal-stack
MIT License
57 stars 12 forks source link

Migration to Containerlab #74

Closed GrigoriyMikhalkin closed 2 years ago

majst01 commented 3 years ago

Nice, does not work in CI because on the CI Machine containerlab is not installed.

majst01 commented 3 years ago

First test on my machine with containerlab v0.16.2 and ignite v0.9.0 gives me:

sudo containerlab deploy --topo mini-lab.clab.yaml                                                                                                                                                                                                                                        
INFO[0000] Parsing & checking topology file: mini-lab.clab.yaml                                                                                                                                                                                                                           
panic: runtime error: invalid memory address or nil pointer dereference                                                                                                                                                                                                                   
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x11e2761]                                                                                                                                                                                                                   

goroutine 1 [running]:                                                                                                                                                                                                                                                                    
github.com/srl-labs/containerlab/runtime/ignite.(*IgniteRuntime).produceGenericContainerList(0xc00024ee70, 0xc0000a2b90, 0x1, 0x1, 0x1, 0x1, 0x0, 0x0, 0xaaa9aaa9aaa9aaa9)                                                                                                                
        /home/runner/work/containerlab/containerlab/runtime/ignite/iginite.go:348 +0x221                                                                                                                                                                                                  
github.com/srl-labs/containerlab/runtime/ignite.(*IgniteRuntime).ListContainers(0xc00024ee70, 0x176ca70, 0xc0004e50e0, 0x0, 0x0, 0x0, 0x0, 0x9, 0x10, 0x0, ...)                                                                                                                           
        /home/runner/work/containerlab/containerlab/runtime/ignite/iginite.go:291 +0x65e                                                                                                                                                                                                  
github.com/srl-labs/containerlab/clab.(*CLab).ListContainers(0xc0000a0cd0, 0x176ca70, 0xc0004e50e0, 0x0, 0x0, 0x0, 0xc00004be80, 0xc0004cd890, 0x503eff, 0xc0003f5a48, ...)                                                                                                               
        /home/runner/work/containerlab/containerlab/clab/clab.go:336 +0x162                                                                                                                                                                                                               
github.com/srl-labs/containerlab/clab.(*CLab).VerifyContainersUniqueness(0xc0000a0cd0, 0x176ca00, 0xc00045d940, 0x0, 0x0)                                                                                                                                                                 
        /home/runner/work/containerlab/containerlab/clab/config.go:426 +0xcd                                                                                                                                                                                                              
github.com/srl-labs/containerlab/clab.(*CLab).CheckTopologyDefinition(0xc0000a0cd0, 0x176ca00, 0xc00045d940, 0xd, 0xc0004cdc08)                                                                                                                                                           
        /home/runner/work/containerlab/containerlab/clab/config.go:349 +0xb0                                                                                                                                                                                                              
github.com/srl-labs/containerlab/cmd.glob..func5(0x1fd7620, 0xc00004b3e0, 0x0, 0x2, 0x0, 0x0)                                                                                                                                                                                             
        /home/runner/work/containerlab/containerlab/cmd/deploy.go:86 +0x2c5                                                                                                                                                                                                               
github.com/spf13/cobra.(*Command).execute(0x1fd7620, 0xc00004b3c0, 0x2, 0x2, 0x1fd7620, 0xc00004b3c0)                                                                                                                                                                                     
        /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:842 +0x472                                                                                                                                                                                                       
github.com/spf13/cobra.(*Command).ExecuteC(0x1fd5940, 0x41ab01, 0x0, 0x0)                                                                                                                                                                                                                 
        /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950 +0x375                                                                                                                                                                                                       
github.com/spf13/cobra.(*Command).Execute(...)                                                                                                                                                                                                                                            
        /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887                                                                                                                                                                                                              
github.com/srl-labs/containerlab/cmd.Execute()                                                                                                                                                                                                                                            
        /home/runner/work/containerlab/containerlab/cmd/root.go:43 +0x2d                                                                                                                                                                                                                  
main.main()                                                                                                                                                                                                                                                                               
        /home/runner/work/containerlab/containerlab/main.go:10 +0x25                                                                                                                                                                                                                      
make: *** [Makefile:53: lab] Error 2

Maybe some version issues ?

majst01 commented 3 years ago

Same with ignite v0.10.0

GrigoriyMikhalkin commented 3 years ago

Same with ignite v0.10.0

I didn't install ignite separately at all. Just tried to run lab with latest version of containerlab(0.16.2) and it worked fine. At least in go.mod they use v0.9.1 version of ignite client lib: https://github.com/srl-labs/containerlab/blob/master/go.mod#L29.

mwindower commented 3 years ago

Removing old ignite VMs with ignite ps -a; ignite rm <ID> solved the problem. @majst01

GrigoriyMikhalkin commented 3 years ago

Removing old ignite VMs with ignite ps -a; ignite rm <ID> solved the problem. @majst01

Ah, i misunderstood the problem. Indeed, ignite caches images(you can list them by running sudo ignite images). I also didn't found a better way but to remove old ones, before trying to test newer versions of image.

GrigoriyMikhalkin commented 3 years ago

Steps to start partition part at the moment: 1) make control-plane-bake 2) make env 3) make lab 4) docker exec [ignite-sandbox-name] ethtool --offload vm_eth0 tx off for both containers 5) docker-compose up --remove-orphans --force-recreate control-plane partition

GrigoriyMikhalkin commented 3 years ago

cc @mwindower, @majst01 Lab is in fully working state. To test it just run make.

One comment regarding implementation. I tried to avoid using vagrant and run each VM in dedicated container image(using virsh). But returned to vagrant(and multiple VMs per container), because it's not only runs virsh, but also does few other things, like creating QEMU disc image. And some other thing that allows to set size label to machine(i didn't figured that out).

IMO, current approach with vagrant and multiple VMs in single container has benefit that it's more easy to implement lab flavors with it(single topology file, single VM config file(Vagrantfile)). Otherwise, we would need to use multiple topology files(one per flavor).

Gerrit91 commented 3 years ago

Also fixing the integration tests would be required.

GrigoriyMikhalkin commented 3 years ago

Work is awesome!

Can you please try to use containerlab from a container? I left some additional comments as well, but overall this looks like quick merge. :D

There's problem when trying to run ignite containers via containerized version of containerlab. The most permissive command that i tried: docker run --rm -it --privileged --network host -v /var/run/docker.sock:/var/run/docker.sock -v /var/run/netns:/var/run/netns -v /etc/hosts:/etc/hosts --pid="host" -v /usr/sbin:/usr/sbin -v /var/lib:/var/lib -v $(pwd):$(pwd) -w $(pwd) ghcr.io/srl-labs/clab containerlab deploy -t mini-lab.clab.yaml

Still gives an error:

ERRO[0000] failed deploy phase for node "leaf1": command ["dmsetup" "create" "--verifyudev" "ignite-d245670ae0ac2c74-base"] exited with "": fork/exec /usr/sbin/dmsetup: no such file or directory

To be clear, container with Vagrant is starting successfully.

GrigoriyMikhalkin commented 3 years ago

images/kernel/lib does include various binary files. If you need those: perhaps use a path from the host system containing the current modules. Those files should not be checked in and also be removed from the git history.

By binaries, do you mean specifically files with .bin extension? I'm not sure if using host system analogs is good idea, because these modules extracted from Cumulus Linux and i suppose they could be somehow modified. But i can experiment with this.

mwindower commented 3 years ago

images/kernel/lib does include various binary files. If you need those: perhaps use a path from the host system containing the current modules. Those files should not be checked in and also be removed from the git history.

By binaries, do you mean specifically files with .bin extension? I'm not sure if using host system analogs is good idea, because these modules extracted from Cumulus Linux and i suppose they could be somehow modified. But i can experiment with this.

I mean all the files under images/kernel/lib. So they are extracted from Cumulus, I see. Then this should reside within a seperate repositoriy.

Gerrit91 commented 2 years ago

Reamining items before releasing mini-lab: