srl-labs / containerlab

container-based networking labs
https://containerlab.dev
BSD 3-Clause "New" or "Revised" License
1.46k stars 249 forks source link

clab config not persistent after redeploy #1685

Open KTodts opened 10 months ago

KTodts commented 10 months ago

I noticed after upgrading my clab version from 43.0 to 47.1 that the config is not persistent after a redeploy of the topology.

Test would be, deploy topology for the first time --> make a config change --> destroy lab --> deploy again --> verify if change is persistent.

1 upgrade clab to version 47.1

root@s4qd-pod75:/home/ubuntu/gitlab/srl-dc-fabric-fundamentals-lab# clab version upgrade
A newer containerlab 0.47.1 is available. Release notes: https://containerlab.dev/rn/0.47.1
You are running containerlab 0.43.0 version
Downloading https://github.com/srl-labs/containerlab/releases/download/v0.47.1/containerlab_0.47.1_linux_amd64.deb
Preparing to install containerlab 0.47.1 from package
(Reading database ... 115507 files and directories currently installed.)
Preparing to unpack .../containerlab_0.47.1_linux_amd64.deb ...
Unpacking containerlab (0.47.1) over (0.43.0) ...
Setting up containerlab (0.47.1) ...

                           _                   _       _
                 _        (_)                 | |     | |
 ____ ___  ____ | |_  ____ _ ____   ____  ____| | ____| | _
/ ___) _ \|  _ \|  _)/ _  | |  _ \ / _  )/ ___) |/ _  | || \
( (__| |_|| | | | |_( ( | | | | | ( (/ /| |   | ( ( | | |_) )
\____)___/|_| |_|\___)_||_|_|_| |_|\____)_|   |_|\_||_|____/

    version: 0.47.1
     commit: 1da5ebd2
       date: 2023-10-24T22:09:40Z
     source: https://github.com/srl-labs/containerlab
 rel. notes: https://containerlab.dev/rn/0.47/#0471

2 Start with a fresh deployment by using --reconfgure

root@s4qd-pod75:/home/ubuntu/gitlab/srl-dc-fabric-fundamentals-lab# ll
total 72
drwxr-xr-x  6 root root 4096 Oct 26 09:31 ./
drwxr-xr-x 17 root root 4096 Oct 26 09:22 ../
-rwxr-xr-x  1 root root   37 Oct 26 09:22 00.show_lab.sh*
-rwxr-xr-x  1 root root   36 Oct 26 09:22 01.deploy_srl.sh*
-rwxr-xr-x  1 root root  335 Oct 26 09:22 03.initialize_hosts.sh*
-rwxr-xr-x  1 root root   49 Oct 26 09:22 10.gnmic_example.sh*
-rwxr-xr-x  1 root root   37 Oct 26 09:22 91.destroy_srl.sh*
-rwxr-xr-x  1 root root  310 Oct 26 09:22 98.check_xdp.sh*
-rwxr-xr-x  1 root root  386 Oct 26 09:22 99.repair_xdp.sh*
drwxr-xr-x  7 root root 4096 Oct 26 09:31 clab-lab/
drwxr-xr-x 10 root root 4096 Oct 26 09:22 config/
drwxr-xr-x  8 root root 4096 Oct 26 09:22 .git/
-rw-r--r--  1 root root   10 Oct 26 09:22 .gitignore
-rw-r--r--  1 root root 1025 Oct 26 09:22 gnmic_subscription.yaml
drwxr-xr-x  2 root root 4096 Oct 26 09:22 img/
-rw-r--r--  1 root root 1192 Oct 26 09:22 README.md
-rw-r--r--  1 root root 1413 Oct 26 09:30 topology.yml
-rw-r--r--  1 root root 1413 Oct 26 09:36 .topology.yml.bak
root@s4qd-pod75:/home/ubuntu/gitlab/srl-dc-fabric-fundamentals-lab# clab dep -t topology.yml --reconfigure
INFO[0000] Containerlab v0.47.1 started
INFO[0000] Parsing & checking topology file: topology.yml
INFO[0000] Removing /home/ubuntu/gitlab/srl-dc-fabric-fundamentals-lab/clab-lab directory...
INFO[0000] Creating lab directory: /home/ubuntu/gitlab/srl-dc-fabric-fundamentals-lab/clab-lab
INFO[0000] Creating container: "host11"
INFO[0000] Creating container: "host10"
INFO[0000] Creating container: "host12"
INFO[0000] Creating container: "host20"
INFO[0000] Creating container: "dc-leaf-1"
INFO[0000] Creating container: "dc-spine-2"
INFO[0000] Creating container: "dc-spine-1"
INFO[0000] Creating container: "dc-leaf-2"
INFO[0002] Creating link: dc-spine-1:e1-2 <--> dc-leaf-2:e1-1
INFO[0003] Creating link: dc-spine-2:e1-2 <--> dc-leaf-2:e1-2
INFO[0003] Creating link: dc-spine-1:e1-1 <--> dc-leaf-1:e1-1
INFO[0003] Creating link: host20:eth1 <--> dc-leaf-2:e1-10
INFO[0004] Creating link: dc-spine-2:e1-1 <--> dc-leaf-1:e1-2
INFO[0004] Creating link: host10:eth1 <--> dc-leaf-1:e1-10
INFO[0004] Creating link: host11:eth1 <--> dc-leaf-1:e1-11
INFO[0004] Creating link: host12:eth1 <--> dc-leaf-1:e1-12
INFO[0005] Running postdeploy actions for Nokia SR Linux 'dc-spine-2' node
INFO[0005] Running postdeploy actions for Nokia SR Linux 'dc-leaf-2' node
INFO[0005] Running postdeploy actions for Nokia SR Linux 'dc-leaf-1' node
INFO[0005] Running postdeploy actions for Nokia SR Linux 'dc-spine-1' node
INFO[0023] Adding containerlab host entries to /etc/hosts file
INFO[0023] Adding ssh config for containerlab nodes
+---+---------------------+--------------+-------------------------------+-------+---------+-----------------+----------------------+
| # |        Name         | Container ID |             Image             | Kind  |  State  |  IPv4 Address   |     IPv6 Address     |
+---+---------------------+--------------+-------------------------------+-------+---------+-----------------+----------------------+
| 1 | clab-lab-dc-leaf-1  | e3236ca45408 | ghcr.io/nokia/srlinux:23.7.1  | srl   | running | 172.20.20.11/24 | 2001:172:20:20::b/64 |
| 2 | clab-lab-dc-leaf-2  | be9c94e81822 | ghcr.io/nokia/srlinux:23.7.1  | srl   | running | 172.20.20.5/24  | 2001:172:20:20::5/64 |
| 3 | clab-lab-dc-spine-1 | ca2ba3def795 | ghcr.io/nokia/srlinux:23.7.1  | srl   | running | 172.20.20.7/24  | 2001:172:20:20::7/64 |
| 4 | clab-lab-dc-spine-2 | 930b7ef359f4 | ghcr.io/nokia/srlinux:23.7.1  | srl   | running | 172.20.20.9/24  | 2001:172:20:20::9/64 |
| 5 | clab-lab-host10     | 7eb4169c5d66 | pklepikov/ubuntu-tools:latest | linux | running | 172.20.20.6/24  | 2001:172:20:20::6/64 |
| 6 | clab-lab-host11     | e36457b5060d | pklepikov/ubuntu-tools:latest | linux | running | 172.20.20.8/24  | 2001:172:20:20::8/64 |
| 7 | clab-lab-host12     | dd238582526d | pklepikov/ubuntu-tools:latest | linux | running | 172.20.20.10/24 | 2001:172:20:20::a/64 |
| 8 | clab-lab-host20     | c9e74db564c3 | pklepikov/ubuntu-tools:latest | linux | running | 172.20.20.12/24 | 2001:172:20:20::c/64 |
+---+---------------------+--------------+-------------------------------+-------+---------+-----------------+----------------------+

3 Log into the node. Make a change, commit and save config. (add default network-instance with a description)

root@s4qd-pod75:/home/ubuntu/gitlab/srl-dc-fabric-fundamentals-lab# ssh admin@clab-lab-dc-leaf-1
Warning: Permanently added 'clab-lab-dc-leaf-1' (ED25519) to the list of known hosts.
admin@clab-lab-dc-leaf-1's password:
Using configuration file(s): []
Welcome to the srlinux CLI.
Type 'help' (and press <ENTER>) if you need any help using this.
--{ [OLD STARTUP] + running }--[  ]--
A:dc-leaf-1# enter candidate
--{ [OLD STARTUP] + candidate shared default }--[  ]--
A:dc-leaf-1# network-instance default description test
--{ [OLD STARTUP] +* candidate shared default }--[  ]--
A:dc-leaf-1# commit now
All changes have been committed. Leaving candidate mode.
--{ [OLD STARTUP] + running }--[  ]--
A:dc-leaf-1# save startup
/system:
    Saved current running configuration as initial (startup) configuration '/etc/opt/srlinux/config.json'

--{ [OLD STARTUP] running }--[  ]--
A:dc-leaf-1#
EOF encountered

Connection to clab-lab-dc-leaf-1 closed.

4 Destroy topology

root@s4qd-pod75:/home/ubuntu/gitlab/srl-dc-fabric-fundamentals-lab# clab des -t topology.yml
INFO[0000] Parsing & checking topology file: topology.yml
INFO[0000] Destroying lab: lab
INFO[0001] Removed container: clab-lab-host11
INFO[0001] Removed container: clab-lab-host20
INFO[0002] Removed container: clab-lab-host12
INFO[0002] Removed container: clab-lab-host10
INFO[0002] Removed container: clab-lab-dc-leaf-2
INFO[0003] Removed container: clab-lab-dc-leaf-1
INFO[0003] Removed container: clab-lab-dc-spine-1
INFO[0003] Removed container: clab-lab-dc-spine-2
INFO[0003] Removing containerlab host entries from /etc/hosts file
INFO[0003] Removing ssh config for containerlab nodes
root@s4qd-pod75:

5 Deploy again. Note that the project directory, clab-lab, is being created again

root@s4qd-pod75:/home/ubuntu/gitlab/srl-dc-fabric-fundamentals-lab# clab dep -t topology.yml
INFO[0000] Containerlab v0.47.1 started
INFO[0000] Parsing & checking topology file: topology.yml
INFO[0000] Creating lab directory: /home/ubuntu/gitlab/srl-dc-fabric-fundamentals-lab/clab-lab
INFO[0000] Creating container: "host20"
INFO[0000] Creating container: "host12"
INFO[0000] Creating container: "host10"
INFO[0000] Creating container: "host11"
INFO[0000] Creating container: "dc-spine-1"
INFO[0000] Creating container: "dc-leaf-1"
INFO[0000] Creating container: "dc-leaf-2"
INFO[0000] Creating container: "dc-spine-2"
INFO[0002] Creating link: dc-spine-2:e1-2 <--> dc-leaf-2:e1-2
INFO[0002] Creating link: dc-spine-1:e1-2 <--> dc-leaf-2:e1-1
INFO[0003] Creating link: host20:eth1 <--> dc-leaf-2:e1-10
INFO[0003] Creating link: dc-spine-1:e1-1 <--> dc-leaf-1:e1-1
INFO[0003] Creating link: dc-spine-2:e1-1 <--> dc-leaf-1:e1-2
INFO[0004] Creating link: host11:eth1 <--> dc-leaf-1:e1-11
INFO[0004] Creating link: host10:eth1 <--> dc-leaf-1:e1-10
INFO[0004] Creating link: host12:eth1 <--> dc-leaf-1:e1-12
INFO[0004] Running postdeploy actions for Nokia SR Linux 'dc-leaf-1' node
INFO[0004] Running postdeploy actions for Nokia SR Linux 'dc-spine-1' node
INFO[0004] Running postdeploy actions for Nokia SR Linux 'dc-spine-2' node
INFO[0004] Running postdeploy actions for Nokia SR Linux 'dc-leaf-2' node
INFO[0022] Adding containerlab host entries to /etc/hosts file
INFO[0022] Adding ssh config for containerlab nodes
+---+---------------------+--------------+-------------------------------+-------+---------+-----------------+----------------------+
| # |        Name         | Container ID |             Image             | Kind  |  State  |  IPv4 Address   |     IPv6 Address     |
+---+---------------------+--------------+-------------------------------+-------+---------+-----------------+----------------------+
| 1 | clab-lab-dc-leaf-1  | 1202a5504106 | ghcr.io/nokia/srlinux:23.7.1  | srl   | running | 172.20.20.9/24  | 2001:172:20:20::9/64 |
| 2 | clab-lab-dc-leaf-2  | e26778959c69 | ghcr.io/nokia/srlinux:23.7.1  | srl   | running | 172.20.20.5/24  | 2001:172:20:20::5/64 |
| 3 | clab-lab-dc-spine-1 | 000173c3b88b | ghcr.io/nokia/srlinux:23.7.1  | srl   | running | 172.20.20.7/24  | 2001:172:20:20::7/64 |
| 4 | clab-lab-dc-spine-2 | f565e19ddaf0 | ghcr.io/nokia/srlinux:23.7.1  | srl   | running | 172.20.20.6/24  | 2001:172:20:20::6/64 |
| 5 | clab-lab-host10     | 6f2d1eb5f3de | pklepikov/ubuntu-tools:latest | linux | running | 172.20.20.11/24 | 2001:172:20:20::b/64 |
| 6 | clab-lab-host11     | 56691d463e2b | pklepikov/ubuntu-tools:latest | linux | running | 172.20.20.12/24 | 2001:172:20:20::c/64 |
| 7 | clab-lab-host12     | 2d61abfc36cc | pklepikov/ubuntu-tools:latest | linux | running | 172.20.20.10/24 | 2001:172:20:20::a/64 |
| 8 | clab-lab-host20     | 65db39a49fff | pklepikov/ubuntu-tools:latest | linux | running | 172.20.20.8/24  | 2001:172:20:20::8/64 |
+---+---------------------+--------------+-------------------------------+-------+---------+-----------------+----------------------+

6 Verify is config change is persistent after redeployment. (default network-instance is not configured)

root@s4qd-pod75:/home/ubuntu/gitlab/srl-dc-fabric-fundamentals-lab# ssh admin@clab-lab-dc-leaf-1
Warning: Permanently added 'clab-lab-dc-leaf-1' (ED25519) to the list of known hosts.
admin@clab-lab-dc-leaf-1's password:
Using configuration file(s): []
Welcome to the srlinux CLI.
Type 'help' (and press <ENTER>) if you need any help using this.
--{ [OLD STARTUP] + running }--[  ]--
A:dc-leaf-1# show network-instance summary
+-----------------------------------------------+------------------------+------------------------+------------------------+-----------------------------------------------+----------------------------------------------------------+
|                     Name                      |          Type          |      Admin state       |       Oper state       |                   Router id                   |                       Description                        |
+===============================================+========================+========================+========================+===============================================+==========================================================+
| mgmt                                          | ip-vrf                 | enable                 | up                     |                                               | Management network instance                              |
+-----------------------------------------------+------------------------+------------------------+------------------------+-----------------------------------------------+----------------------------------------------------------+
hellt commented 10 months ago

Thanks for a detailed report @KTodts we will have a look at it

steiler commented 10 months ago

I can not confirm this. The message Creating lab directory: ... is expected, we call the os.MakeAll functions, that takes a path and creates all the dirs along the path that do not exist, like mkdirwith the -p option. So if the path towards the lab folder is there it is basically a noop. Deploy:

~/projects/containerlab4$ sudo clab deploy -t lab-examples/srl01/
INFO[0000] Containerlab v0.47.2 started
INFO[0000] Parsing & checking topology file: srl01.clab.yml
INFO[0000] Creating docker network: Name="clab", IPv4Subnet="172.20.20.0/24", IPv6Subnet="2001:172:20:20::/64", MTU='ל'
INFO[0000] Creating lab directory: /home/steiler/projects/containerlab4/clab-srl01
INFO[0000] Creating container: "srl"
INFO[0001] Running postdeploy actions for Nokia SR Linux 'srl' node
INFO[0022] Adding containerlab host entries to /etc/hosts file
INFO[0022] Adding ssh config for containerlab nodes
+---+----------------+--------------+-----------------------+------+---------+----------------+----------------------+
| # |      Name      | Container ID |         Image         | Kind |  State  |  IPv4 Address  |     IPv6 Address     |
+---+----------------+--------------+-----------------------+------+---------+----------------+----------------------+
| 1 | clab-srl01-srl | af0a98f1c3b0 | ghcr.io/nokia/srlinux | srl  | running | 172.20.20.2/24 | 2001:172:20:20::2/64 |
+---+----------------+--------------+-----------------------+------+---------+----------------+----------------------+

Configure:

steiler@vbox:~/projects/containerlab4$ ssh clab-srl01-srl
Warning: Permanently added 'clab-srl01-srl' (ED25519) to the list of known hosts.
................................................................
:                  Welcome to Nokia SR Linux!                  :
:              Open Network OS for the NetOps era.             :
:                                                              :
:    This is a freely distributed official container image.    :
:                      Use it - Share it                       :
:                                                              :
: Get started: https://learn.srlinux.dev                       :
: Container:   https://go.srlinux.dev/container-image          :
: Docs:        https://doc.srlinux.dev/22-11                   :
: Rel. notes:  https://doc.srlinux.dev/rn22-11-1               :
: YANG:        https://yang.srlinux.dev/release/v22.11.1       :
: Discord:     https://go.srlinux.dev/discord                  :
: Contact:     https://go.srlinux.dev/contact-sales            :
................................................................

Using configuration file(s): []
Welcome to the srlinux CLI.
Type 'help' (and press <ENTER>) if you need any help using this.
--{ running }--[  ]--
A:srl# enter candidate
--{ candidate shared default }--[  ]--
A:srl# set network-instance default description "my funny description"
--{ * candidate shared default }--[  ]--
A:srl# set network-instance default admin-state enable
--{ * candidate shared default }--[  ]--
A:srl# commit now save
/system:
    Saved current running configuration as initial (startup) configuration '/etc/opt/srlinux/config.json'

All changes have been committed. Leaving candidate mode.
--{ running }--[  ]--
A:srl# quit
Connection to clab-srl01-srl closed.

Check:

steiler@vbox:~/projects/containerlab4$ grep -B3 -A2 funny clab-srl01/srl/config/config.json
    {
      "name": "default",
      "admin-state": "enable",
      "description": "my funny description"
    },
    {

Destroy:

steiler@vbox:~/projects/containerlab4$ sudo clab destroy -t lab-examples/srl01/
INFO[0000] Parsing & checking topology file: srl01.clab.yml
INFO[0000] Destroying lab: srl01
INFO[0000] Removed container: clab-srl01-srl
INFO[0000] Removing containerlab host entries from /etc/hosts file
INFO[0000] Removing ssh config for containerlab nodes

Redeploy:

steiler@vbox:~/projects/containerlab4$ sudo clab deploy -t lab-examples/srl01/
INFO[0000] Containerlab v0.47.2 started
INFO[0000] Parsing & checking topology file: srl01.clab.yml
INFO[0000] Creating docker network: Name="clab", IPv4Subnet="172.20.20.0/24", IPv6Subnet="2001:172:20:20::/64", MTU='ל'
INFO[0000] Creating lab directory: /home/steiler/projects/containerlab4/clab-srl01
INFO[0000] Creating container: "srl"
INFO[0000] Running postdeploy actions for Nokia SR Linux 'srl' node
INFO[0016] Adding containerlab host entries to /etc/hosts file
INFO[0016] Adding ssh config for containerlab nodes
+---+----------------+--------------+-----------------------+------+---------+----------------+----------------------+
| # |      Name      | Container ID |         Image         | Kind |  State  |  IPv4 Address  |     IPv6 Address     |
+---+----------------+--------------+-----------------------+------+---------+----------------+----------------------+
| 1 | clab-srl01-srl | 003c92e9f56e | ghcr.io/nokia/srlinux | srl  | running | 172.20.20.2/24 | 2001:172:20:20::2/64 |
+---+----------------+--------------+-----------------------+------+---------+----------------+----------------------+

Check file:

steiler@vbox:~/projects/containerlab4$ grep -B3 -A2 funny clab-srl01/srl/config/config.json
    {
      "name": "default",
      "admin-state": "enable",
      "description": "my funny description"
    },
    {

Check SRL:

steiler@vbox:~/projects/containerlab4$ ssh clab-srl01-srl
Warning: Permanently added 'clab-srl01-srl' (ED25519) to the list of known hosts.
................................................................
:                  Welcome to Nokia SR Linux!                  :
:              Open Network OS for the NetOps era.             :
:                                                              :
:    This is a freely distributed official container image.    :
:                      Use it - Share it                       :
:                                                              :
: Get started: https://learn.srlinux.dev                       :
: Container:   https://go.srlinux.dev/container-image          :
: Docs:        https://doc.srlinux.dev/22-11                   :
: Rel. notes:  https://doc.srlinux.dev/rn22-11-1               :
: YANG:        https://yang.srlinux.dev/release/v22.11.1       :
: Discord:     https://go.srlinux.dev/discord                  :
: Contact:     https://go.srlinux.dev/contact-sales            :
................................................................

Using configuration file(s): []
Welcome to the srlinux CLI.
Type 'help' (and press <ENTER>) if you need any help using this.
--{ running }--[  ]--
A:srl# info from running network-instance default
    network-instance default {
        admin-state enable
        description "my funny description"
    }
--{ running }--[  ]--
A:srl#

So, well... someone else maybe also try this, but I can't reproduce the issue.

KTodts commented 10 months ago

I don't know if it makes any difference but I'm using startup-config in the topology file.

edit: My understanding was, that clab project dir has priority to the startup-config file. When you deploy for the first time there is no clab project dir, so it takes the config from startup-config. Once you alter the config and redeploy it should take the config from the clab project dir

steiler commented 10 months ago

Hmm @hellt whats our definition of a startup-config? Is that the if no config already exists startup-config or is it the always and in any case startup-config?

steiler commented 10 months ago

ok, if a config is present in the nodes lab dir we should use that, if not e.g. --reconfigure we should download. I'll figure out how we do that best

hellt commented 10 months ago

ah, startup-config takes precendence over the local dir @KTodts The goal of a startup-config is to be the "startup config".

The idea was that our topology should be as declarative as possible, so if we have a startup config in the topo file, this file should be used as a startup config, and thus overriding any config present in the node lab directory.

hellt commented 10 months ago

Although this note here https://containerlab.dev/manual/nodes/#embedded-startup-config says otherwise :D

To be honest, I think we need to then use existing labdir config indeed

/cc @steiler @KTodts

KTodts commented 10 months ago

Although this note here https://containerlab.dev/manual/nodes/#embedded-startup-config says otherwise :D

To be honest, I think we need to then use existing labdir config indeed

/cc @steiler @KTodts

This way after installing a project from a repo you have the possibility to build on top of that.

steiler commented 10 months ago

Yes and the note says so... We do have the enforce-startup-config flag otherwise... This behavior got screwed when I implemented the http(s) download of the config... Mea culpa. Will fix it!

gusman commented 8 months ago

I just found this issue has been occurred starting from version 0.45.0.

The main issue that I have experienced is even though I have saved my config before destroying the topology, the config is not persistent anymore. After executing deploy command it always copies or uses the config file from startup-config definition in topo-yaml file.,

hellt commented 8 months ago

yes, this is still unresolved. I will link your internal message to be evaluated when we get to fix this

mahmutaydin1 commented 4 months ago

@hellt I am having same issue. I have startup config in clab lab directory. When I deploy lab again, startup config is not seen. have you any progress for this bug

thank you