Dockerize IOU

[ghost]

Cisco IOU is quite complicate to integrate, it needs 32-bit shared libraries, a special NETMAP configuration and iouyap. Using docker images for IOU would encapsulate all this special stuff. Julien made an early test with IOU within docker, see https://github.com/GNS3/gns3-registry/tree/master/docker/iou.

I've created a more advanced version, see https://github.com/ehlers/gns3-docker-images/tree/master/iou-base. It consists of a base image, that contains everything except the IOU binary, the license and the initial config. With the help of the create-iou-image script the user creates a new docker IOU image for each IOU binary, adding the missing stuff (binary license, config) to the base. As the base image is quite small (based on alpine, about 15 MB), this is a quite efficient way using IOU.

Of course the use of create-iou-image is just a temporary solution, a much better way would be a dialog integrated into GNS3.

During the development I encountered the following issues:

• Only support for ethernet interfaces
  Especially for training this is a mayor drawback.
• The base MAC address of an IOU device is based on the device ID, see https://github.com/GNS3/gns3-server/issues/557. That means, that every IOU device in a project needs a unique ID. Currently the docker image has no way choosing one. Currently I'm using the device name to create an ID between 1..997, but that's not safe.
  The GNS3 GUI already creates device IDs. If those IDs can be sent to the docker VMs, this issue should be solved. Maybe even https://github.com/GNS3/gns3-server/issues/557 can be solved that way.
• The interface names differ between IOU and the GNS3 GUI. That may create some confusion. The other VM types have a flexible way naming the interfaces in the GUI. Maybe that can be also added to the docker container.
• There is no GUI field in docker to modify the NVRAM and MEM size. I'm using environment variables for the (few) situations, where the defaults needs to be changed.

While IOU in docker basically works, a nice support needs some (small) changes. For me the main issue is the loss of the serial interfaces.

@noplay @grossmj What do you think about IOU within docker?

[julien-duponchelle]

I think it's how we will support IOU in the future. We know that one day the 32 bit support on most linux will be dropped and we will need to have an alternative. The deadline is ubuntu 18.10.

Your alpine image is very interesting due to the small size.

I see two path:

• IOU is a container like other in the interface. You create them using gns3a, no more dedicated IOU settings / wizard. And we improve a lot docker support to allow container to have custom settings
• IOU is a different interface

We don't know how IOL in VIRL will work it can change the strategy.

• Only support for ethernet interfaces Especially for training this is a mayor drawback.

Jeremy need to confirm that. But I think serial is just display. Under the hood the network communication is the same.

We can probably modify the docker support to inject ethernet adapter with a different name in the container to symbolize that this adapter are serial link.

Question is it specific to IOU or could we use that in other container.

Also it seem IOL doesn't support serial.

• The base MAC address of an IOU device is based on the device ID, see GNS3/gns3-server#557. That means, that every IOU device in a project needs a unique ID. Currently the docker image has no way choosing one. Currently I'm using the device name to create an ID between 1..997, but that's not safe. The GNS3 GUI already creates device IDs. If those IDs can be sent to the docker VMs, this issue should be solved. Maybe even GNS3/gns3-server#557 can be solved that way.

No idea on this topic :(

• The interface names differ between IOU and the GNS3 GUI. That may create some confusion. The other VM types have a flexible way naming the interfaces in the GUI. Maybe that can be also added to the docker container.

I will start some work on port naming architecture https://github.com/GNS3/gns3-server/issues/676 We can probably add a support for docker in a futur release.

• There is no GUI field in docker to modify the NVRAM and MEM size. I'm using environment variables for the (few) situations, where the defaults needs to be changed.

Perhaps we need a settings system for containers. When you import the GNS3A the appliance file say this variable can be configured and this will be reflect to the container env.

Also I'm not a big fan of embedding the IOU image in the container because it's break the project portability. I think we could find a way for mounting the IOU images directory and with a env variable choose the correct image.

But your technique work with today GNS3 and that's cool. On a linux box you can use IOU without the VM or ugly 32 bits dependencies just docker.

[ghost]

Thanks for your kind review.

I think it's how we will support IOU in the future. We know that one day the 32 bit support on most linux will be dropped and we will need to have an alternative. The deadline is ubuntu 18.10.

The docker solution depends on the kernel's capability to execute 32 bit binaries even on a 64 bit system. As long as this functionality is kept, we are safe. But when this is dropped, only a VM solution will help.

[julien-duponchelle]

I doubt kernel will stop support 32 bit. But the package system could stop. If the container still use an old distribution no issue. And I think docker work on stuff for multi architecture support. Even if CPU stop supporting 32 bit or everybody start to use an ARM processor we could imagine using that.

On Tue, Sep 13, 2016 at 9:40 AM Bernhard Ehlers notifications@github.com wrote:

Thanks for your kind review.

I think it's how we will support IOU in the future. We know that one day the 32 bit support on most linux will be dropped and we will need to have an alternative. The deadline is ubuntu 18.10.

The docker solution depends on the kernel's capability to execute 32 bit binaries even on a 64 bit system. As long as this functionality is kept, we are safe. But when this is dropped, only a VM solution will help.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/GNS3/gns3-gui/issues/1508#issuecomment-246600904, or mute the thread https://github.com/notifications/unsubscribe-auth/AAVFXUMSmD1P_tuJlaHtOjUrtTc-doYzks5qplNpgaJpZM4J7IRK .

[ghost]

Serial links work, sort-of !!!

I've connected an IOU serial link via iouyap to an docker ethernet interface, connected this ethernet interface to another docker IOU, where the IOU serial link is connected to an docker ethernet interface. They can communicate via this link.

Then I started wireshark. It tries to decode the serial data as an ethernet frame, what (of course) fails. When I compare this trace with a trace of a real serial link, the raw data is OK. Even packet with data length < 64 byte (minimum ethernet frame length) are fine. Then I saved the capture, changed the encapsulation with "editcap -T chdlc infile outfile" and loaded it again. This results in a perfectly fine trace file.

So technically it's possible to use the serial interface. The main task is, to let the GUI handle this ethernet link as a serial one.

[julien-duponchelle]

Great news.

[ghost]

Updated my iou-base docker image, now you can control the number of serial interfaces with the environment variable SERIAL.

On the server side, the only issue left is that of the unique device ID.

On the GUI the setting of environment variables could be handled more user-friendly for all docker images. That's not IOU specific. But the interface handling within the GUI, especially the serial interfaces, is special for IOU.

[grossmj]

I think we going to have to have something like this ssooner or later.

@b-ehlers do you have your repository https://github.com/ehlers/gns3-docker-images/tree/master/iou-base available somewhere?

[ghost]

Here the files from the iou-base directory: iou-base.tar.gz Update: replaced the Buster based libc by a libc based on Stretch, see https://github.com/GNS3/gns3-gui/issues/1508#issuecomment-936629023.

Please note, that I haven't updated them for quite a while, currently it doesn't work and needs fixing (not me).

• It depends on iouyap, that is no longer available in alpine. I wasn't able to use ubridge instead, as I found no way to create an IOU bridge from the command line.
• The Dockerfile loads the IOU utils from https://git.b-ehlers.de/ehlers/IOUtools/raw/branch/master/, which is no longer available. The utils are an official part of GNS3, the latest version is available` in https://github.com/GNS3/gns3-server/tree/3.0/gns3server/compute/iou/utils/

[ghost]

Just made some tests with Fedora 34 Workstation. Now I'm no longer sure, that the isolation provided by Docker is sufficient.

Test 1: Debian container on Debian host, everything is fine

$ docker run -ti -v /home/behlers:/user debian
root@a08558b66a75:/# cd
root@a08558b66a75:~# apt-get update
*** output removed
root@a08558b66a75:~# apt-get install python3
*** output removed
root@a08558b66a75:~# printf '\0\0\0\0' > /etc/hostid
root@a08558b66a75:~# python3 /user/GNS3/IOU/CiscoIOUKeygen.py | grep -A1 license > .iourc
root@a08558b66a75:~# cat .iourc 
[license]
a08558b66a75 = 63992d5788e7c312;
root@a08558b66a75:~# /user/GNS3/images/IOU/x86_64bi_linux-l3-adventerprise-ms.154-2.S.bin 1
IOS On Unix - Cisco Systems confidential, internal use only
netio error: unable to open NETMAP: No such file or directory
*** .iourc verifies OK, if a NETMAP would be present the image would start
root@a08558b66a75:~# /user/GNS3/images/IOU/i86bi-linux-l2-adventerprisek9-15.1a.bin 1
bash: /user/GNS3/images/IOU/i86bi-linux-l2-adventerprisek9-15.1a.bin: No such file or directory
*** As no 32 bit library is installed, this error is expected
root@a08558b66a75:~# tar xvfz /user/GNS3/IOU/libc-i386.tar.gz -C /
*** output removed
root@a08558b66a75:~# /user/GNS3/images/IOU/i86bi-linux-l2-adventerprisek9-15.1a.bin 1
***************************************************************
IOS On Unix - Cisco Systems confidential, internal use only
netio error: unable to open NETMAP: No such file or directory
*** .iourc verifies OK, if a NETMAP would be present the image would start
root@a08558b66a75:~#

Test 2: Fedora container on Debian host

Same commands as above, just with the fedora image. The "apt-get" commands will be skipped, as the image already contains python3 (and fedora uses DNF).

This works absolutely fine, no difference to the debian image.

Test 3: Debian container on Fedora 34 host, the 32 bit IOU image is not able to verify the license

$ docker run -ti --security-opt label=disable -v /home/user:/user debian
root@b9f3e938db97:/# cd
root@b9f3e938db97:~# apt-get update
*** output removed
root@b9f3e938db97:~# apt-get install python3
*** output removed
root@b9f3e938db97:~# printf '\0\0\0\0' > /etc/hostid
root@b9f3e938db97:~# python3 /user/CiscoIOUKeygen.py | grep -A1 license > .iourc
root@b9f3e938db97:~# /user/x86_64bi_linux-l3-adventerprise-ms.154-2.S.bin 1
IOS On Unix - Cisco Systems confidential, internal use only
netio error: unable to open NETMAP: No such file or directory
*** .iourc verifies OK, if a NETMAP would be present the image would start
root@b9f3e938db97:~# /user/i86bi-linux-l2-adventerprisek9-15.1a.bin 1
bash: /user/i86bi-linux-l2-adventerprisek9-15.1a.bin: No such file or directory
*** As no 32 bit library is installed, this error is expected
root@b9f3e938db97:~# tar xvfz /user/libc-i386.tar.gz -C /
*** output removed
root@b9f3e938db97:~# /user/i86bi-linux-l2-adventerprisek9-15.1a.bin 1
***************************************************************
IOS On Unix - Cisco Systems confidential, internal use only

IOU License Error: host not found in iourc file
License for key 373 required on host "b9f3e938db97". 
Obtain a license for this key and host from the following location:

http://wwwin-enged.cisco.com/ios/iou/license/index.html

Place in your iourc file as follows (see also the web page
for further details on iourc file format and location)

[license]
b9f3e938db97 = <16 char license>;

root@b9f3e938db97:~#

Test 4: IOU image natively on Fedora

Same result as on Test 3, the 64bit IOU image works, but the 32bit image fails to verify the license

Conclusion

I have no real idea, why this tests fail on Fedora 34. I would expect, that at least the Debian image within Docker would work. As a docker image includes everything except the kernel, my only idea is, that the Fedora 34 kernel is incompatible to IOU. But that means, that docker provides not enough isolation to run IOU on every Linux OS.

[grossmj]

@b-ehlers strange, thanks for the heads up 👍

[ghost]

Good news, it's now working, also on Fedora 34.

The 32-bit libc I was using (and that is included in the previously attached iou-base.tar.gz) was created from Debian Buster. This version is running fine here on my Debian systems.

I created another 32-bit libc based on Debian Stretch and that works fine both on Fedora and on Debian: libc-i386-stretch.tar.gz If we experience further problems we might need to use an even older Debian or Ubuntu distribution as a base.

My theory, what happened: Some syscalls to the kernel got some slight changes over time. The IOU images are quite old and are very likely compiled against an old libc, that was using some older syscalls. But the libc from Buster uses newer syscalls. This mismatch causes the IOU image to fail. The libc from Stretch seems to use the syscalls in a way the IOU binaries expect.

[ghost]

From the initial comment:

Cisco IOU is quite complicate to integrate, it needs 32-bit shared libraries, a special NETMAP configuration and iouyap. Using docker images for IOU would encapsulate all this special stuff.

Interface handling with NETMAP and iouyap/ubridge was done a long time ago and works flawless. So we won't gain much moving this into a Docker image. Only dealing with the 32-bit shared libraries would be simplified by using Docker.

I got two alternative ideas, that both allow to use our own shared libraries (those from Debian Stretch are working well with IOU). Both options are very easy to implement.

1. Bubblewrap

Bubblewrap creates a sandbox, which allows us to remap some subdirectories like /lib to user defined directories. That allows us to use our own shared libraries, independantly of the system provided libraries. When starting an IOU image the command has to be prefixed by bwrap command line arguments, all other aspects of the IOU images remain the same as before. So only very few changes in GNS3 are needed, mainly the subprocess calls in iou_vm.py need to be adapted. I created a basic patch and it works well.

But this adds an additional dependancy. On most systems the bubblewrap installation is simple, but sometimes it's not. The README.Debian is a good source of information. Even though it works fine on my Debian 11, I don't recommend it. I fear, that it may create more issues than it solves.

2. Use a user defined loader with user defined libraries

Normally, when starting a binary, the system looks in the ELF header, gets the loader information and starts it. But you can directly start the loader with the program as its argument. For example /lib64/ld-linux-x86-64.so.2 /usr/bin/ls uses the system loader /lib64/ld-linux-x86-64.so.2 to load and run the ls program. By using our user defined loader, we can run programs without using the system loader. By using the loader option --library-path PATH, we can also use our own shared libraries.

This option will not create any isolation for the IOU images, but uses loader and shared libraries independantly from the system. It works even on Alpine, that normally doesn't support GLIBC libraries. Again, in GNS3 mainly the subprocess calls have to be adapted. I created a patch for iou_vm.py and it works fine. As it works in user mode and no additional software needs to be installed, this is my prefered solution.

If there is interest, I am happy to provide more information.

[grossmj]

Looks great, I also prefer the 2nd solution. Please send your patched iou_vm.py, thanks 👍

[ghost]

The patch was made against GNS3 v2.2.33, but I suggest to port it to v3.x to keep 2.2 stable.

As a nice feature it falls back to the system loader/libraries, when it doesn't find the user defined loader.

Here my patch for the second solution:

diff --git a/gns3server/compute/iou/iou_vm.py b/gns3server/compute/iou/iou_vm.py
index 617b247e..cfa364fb 100644
--- a/gns3server/compute/iou/iou_vm.py
+++ b/gns3server/compute/iou/iou_vm.py
@@ -80,6 +80,8 @@ class IOUVM(BaseNode):
         self._started = False
         self._nvram_watcher = None
         self._path = self.manager.get_abs_image_path(path, project.path)
+        self._lib_base = self.manager.get_images_directory()
+        self._loader = None
         self._license_check = True

         # IOU settings
@@ -142,6 +144,7 @@ class IOUVM(BaseNode):
         """

         self._path = self.manager.get_abs_image_path(path, self.project.path)
+        self._loader = None
         log.info('IOU "{name}" [{id}]: IOU image updated to "{path}"'.format(name=self._name, id=self._id, path=self._path))

     @property
@@ -173,8 +176,9 @@ class IOUVM(BaseNode):
         Finds the default RAM and NVRAM values for the IOU image.
         """

+        self._check_requirements()
         try:
-            output = await gns3server.utils.asyncio.subprocess_check_output(self._path, "-h", cwd=self.working_dir, stderr=True)
+            output = await gns3server.utils.asyncio.subprocess_check_output(*self._loader, self._path, "-h", cwd=self.working_dir, stderr=True)
             match = re.search(r"-n <n>\s+Size of nvram in Kb \(default ([0-9]+)KB\)", output)
             if match:
                 self.nvram = int(match.group(1))
@@ -216,6 +220,20 @@ class IOUVM(BaseNode):
         if not os.access(self._path, os.X_OK):
             raise IOUError("IOU image '{}' is not executable".format(self._path))

+        # set loader command
+        if elf_header_start[4] == 1:
+            loader = os.path.join(self._lib_base, "lib", "ld-linux.so.2")
+            lib_path = ":".join((os.path.join(self._lib_base, "lib"),
+                                 os.path.join(self._lib_base, "lib", "i386-linux-gnu")))
+        else:
+            loader = os.path.join(self._lib_base, "lib64", "ld-linux-x86-64.so.2")
+            lib_path = ":".join((os.path.join(self._lib_base, "lib64"),
+                                 os.path.join(self._lib_base, "lib", "x86_64-linux-gnu")))
+        if os.path.isfile(loader):
+            self._loader = [loader, "--library-path", lib_path]
+        else:
+            self._loader = []
+
     def __json__(self):

         iou_vm_info = {"name": self.name,
@@ -378,8 +396,10 @@ class IOUVM(BaseNode):
         Checks for missing shared library dependencies in the IOU image.
         """

+        env = os.environ.copy()
+        env["LD_TRACE_LOADED_OBJECTS"] = "1"
         try:
-            output = await gns3server.utils.asyncio.subprocess_check_output("ldd", self._path)
+            output = await gns3server.utils.asyncio.subprocess_check_output(*self._loader, self._path, env=env)
         except (OSError, subprocess.SubprocessError) as e:
             log.warning("Could not determine the shared library dependencies for {}: {}".format(self._path, e))
             return
@@ -551,7 +571,7 @@ class IOUVM(BaseNode):
                 log.info("Starting IOU: {}".format(command))
                 self.command_line = ' '.join(command)
                 self._iou_process = await asyncio.create_subprocess_exec(
-                    *command,
+                    *self._loader, *command,
                     stdout=asyncio.subprocess.PIPE,
                     stdin=asyncio.subprocess.PIPE,
                     stderr=subprocess.STDOUT,
@@ -1070,7 +1090,7 @@ class IOUVM(BaseNode):
         if "IOURC" not in os.environ:
             env["IOURC"] = self.iourc_path
         try:
-            output = await gns3server.utils.asyncio.subprocess_check_output(self._path, "-h", cwd=self.working_dir, env=env, stderr=True)
+            output = await gns3server.utils.asyncio.subprocess_check_output(*self._loader, self._path, "-h", cwd=self.working_dir, env=env, stderr=True)
             if re.search(r"-l\s+Enable Layer 1 keepalive messages", output):
                 command.extend(["-l"])
             else:

There are several decision to make before using this solution:

• Where should the loader and library be stored? My patch uses the lib and lib64 subdirectories of the IOU image directory.
• Will GNS3 provide a package with these libraries? If yes, from what distribution. In my tests Debian 9 (Stretch) and CentOS 7 are working fine. But newer distribution fail, when using them within Fedora 36. But Debian 9 is already EOL and CentOS 7 will be in 1 3/4 years. For Debian 9 there is a commercial offering for another 5 years of support, the Debian LTS/Extended page says that "updates and security fixes will be available for all Debian users without cost". I haven't checked the Ubuntu distributions.

Here the libraries for Debian 9 Stretch, including the build scripts (using Docker): libc_iou_stretch.tar.gz. Within this archive it contains libc-iou.tar.gz. Unpack it into the IOU image directory to use it with this patch.

[ghost]

Extended solution: An additional check in _check_requirements() tests, if the user defined loader is really suitable for the IOU binary. This is not really required, it's just an extra safeguard.

Here the complete patch (against GNS3 2.2.33): iou_loader.patch

[grossmj]

I have created a PR with your patch: https://github.com/GNS3/gns3-server/pull/2086 and tested on Ubuntu 22.04 using your provided Debian 9 Stretch libraries, it worked fine :)

Where should the loader and library be stored? My patch uses the lib and lib64 subdirectories of the IOU image directory.

This location could work. It could also be in the ~/.config/GNS3/3.0 directory but I think I prefer the IOU image directory.

Will GNS3 provide a package with these libraries?

Eventually yes I think. For now we are safe as long as 32-bit support isn't dropped, right?

[ghost]

Will GNS3 provide a package with these libraries?

Eventually yes I think. For now we are safe as long as 32-bit support isn't dropped, right?

Correct, this patch will fallback to the system loader/libraries, when it doesn't find the user defined loader. So as long a distribution supports 32 bit glibc libraries, that are not too recent, it's safe to use them. Notably exceptions are Alpine Linux (that doesn't provide glibc libraries) and recent Fedora distibutions (their 32-bit libraries are too new for IOU, see https://github.com/GNS3/gns3-server/issues/1979#issuecomment-938111829).

But in any case the linux kernel has to support 32-bit programs, otherwise IOUs won't run. Even Docker won't help then. But I assume it will take very long until Linux drops 32-bit support.

[mmaeso]

CML 2.7 has added support for x86_64 based IOU running IOS XE 17. The images run without (apparent) issues on an Ubuntu 22.04 docker container, since they don't even require an iourc file to run. Considering other vendors are adding support for containerized network devices (cRPD, XRd, cSRX, etc), would this be of any interest to the development team?