search

GNS3 / gns3-registry

GNS3 devices registry
GNU General Public License v3.0
323 stars 387 forks source link

IOU VM "IOU1" process has stopped with return code: -6 (IOU L2 v17.12.1) #868

Closed Zero3Dash closed 2 months ago

Zero3Dash commented 3 months ago

https://github.com/GNS3/gns3-registry/blob/master/appliances/cisco-iou-l2.gns3a

The Cisco IOU L2 image (IOU L2 v17.12.1) for x86_64_crb_linux_l2-adventerprisek9-ms presents IOU VM "IOU1" process has stopped with return code: -6 when attempting to run with the default options.

Zero3Dash commented 3 months ago

Running on VMware ESXi Server version 2.2.46

grossmj commented 3 months ago

The Cisco IOU L2 image (IOU L2 v17.12.1) for x86_64_crb_linux_l2-adventerprisek9-ms presents IOU VM "IOU1" process has stopped with return code: -6 when attempting to run with the default options.

Maybe more RAM is needed? @Desposyni do you know?

Zero3Dash commented 3 months ago

I've tried 256MB, 512MB , 768MB and 1028MB. Same result.

The layer 3 (router) image of the same version number works fine with 256 MB.

Desposyni commented 3 months ago

It's not the RAM, the L2-IOU only consumes around 150MB when idle, and as long as the 'use default memory' is checked the IOU file will choose to use 768MB as its max. It's something with the GNS3 VM. The L2-IOU works with Mint and Pop!_OS. Not sure if it's the kernel or something else within the distro. When running the files directly with -d and -N flags, the L3-IOU and L2-IOU throw the same warnings, but L2 aborts. The only difference is the L3-IOU detects a platform of IOL, while the L2-IOU ends up with a platform of (null). The first 7 bytes of the file show it's a 64 bit ELF version 1 executable and the file name doesn't start or end with the usual IOU file name, so it also has issues being detected as a legitimate IOU file. I tried editing images.py on my server to make it detect, but no luck. Any ideas as to what dependency the VM is missing that other OSes have and that the L2 needs, but the L3 doesn't?

Zero3Dash commented 3 months ago

I've built a new GNS3 VM using Ubuntu 22.04.04 (fully updated) and GNS3 Server 2.2.46 and the image boots fine.

sysadmin@gns3vm:~$ gns3server No configuration file could be found or read 2024-03-30 10:49:53 INFO run.py:218 GNS3 server version 2.2.46 2024-03-30 10:49:53 INFO run.py:220 Copyright (c) 2007-2024 GNS3 Technologies Inc. 2024-03-30 10:49:53 INFO run.py:242 Running with Python 3.10.12 and has PID 1251 2024-03-30 10:49:53 INFO run.py:248 Using system certificate store for SSL connections 2024-03-30 10:49:53 INFO run.py:78 Current locale is en_US.UTF-8 2024-03-30 10:49:53 INFO web_server.py:338 Starting server on 0.0.0.0:3080 2024-03-30 10:49:53 INFO init.py:70 Load controller configuration file /home/sysadmin/.config/GNS3/2.2/gns3_controller.conf 2024-03-30 10:49:53 INFO init.py:74 Controller is starting 2024-03-30 10:49:53 INFO init.py:314 Installing base configs in '/home/sysadmin/GNS3/configs' 2024-03-30 10:49:53 INFO init.py:327 Installing built-in disks in '/home/sysadmin/GNS3/images/QEMU' 2024-03-30 10:49:53 INFO init.py:252 Comparing controller version 2.2.46 with config version 2.2.46 2024-03-30 10:49:53 INFO compute.py:64 Create compute local 2024-03-30 10:49:53 INFO compute.py:364 Connecting to compute 'local' 2024-03-30 10:49:53 INFO web_log.py:211 127.0.0.1 [30/Mar/2024:10:49:53 +0000] "GET /v2/compute/capabilities HTTP/1.1" 200 553 "-" "Python/3.10 aiohttp/3.9.3" 2024-03-30 10:49:53 INFO web_server.py:247 Computing image checksums... 2024-03-30 10:49:53 INFO notification_handler.py:50 New client has connected to compute WebSocket 2024-03-30 10:49:53 INFO web_server.py:249 Finished computing image checksums 2024-03-30 10:49:53 INFO compute.py:431 Connected to compute 'local' WebSocket 'http://127.0.0.1:3080/v2/compute/notifications/ws' 2024-03-30 10:49:57 INFO web_log.py:211 192.168.1.50 [30/Mar/2024:10:49:57 +0000] "GET /v2/version HTTP/1.1" 200 229 "-" "GNS3 QT Client v2.2.46" 2024-03-30 10:49:57 INFO notification_handler.py:73 New client has connected to controller WebSocket 2024-03-30 10:49:57 INFO web_log.py:211 192.168.1.50 [30/Mar/2024:10:49:57 +0000] "GET /v2/templates HTTP/1.1" 200 3976 "-" "GNS3 QT Client v2.2.46" 2024-03-30 10:49:57 INFO web_log.py:211 192.168.1.50 [30/Mar/2024:10:49:57 +0000] "GET /v2/computes HTTP/1.1" 200 1022 "-" "GNS3 QT Client v2.2.46" 2024-03-30 10:49:57 INFO web_log.py:211 192.168.1.50 [30/Mar/2024:10:49:57 +0000] "GET /v2/projects HTTP/1.1" 200 832 "-" "GNS3 QT Client v2.2.46" 2024-03-30 10:49:57 INFO web_log.py:211 192.168.1.50 [30/Mar/2024:10:49:57 +0000] "GET /v2/version HTTP/1.1" 200 229 "-" "GNS3 QT Client v2.2.46" 2024-03-30 10:49:57 INFO web_log.py:211 192.168.1.50 [30/Mar/2024:10:49:57 +0000] "GET /v2/appliances?symbol_theme=Affinity-circle-blue HTTP/1.1" 200 999047 "-" "GNS3 QT Client v2.2.46" 2024-03-30 10:49:57 INFO project.py:78 Project 9ac13950-01c8-492b-83ac-fb65fb205c75 with path '/home/sysadmin/GNS3/projects/9ac13950-01c8-492b-83ac-fb65fb205c75' created 2024-03-30 10:49:57 INFO web_log.py:211 127.0.0.1 [30/Mar/2024:10:49:57 +0000] "POST /v2/compute/projects HTTP/1.1" 201 300 "-" "Python/3.10 aiohttp/3.9.3" 2024-03-30 10:49:57 INFO iou_vm.py:73 IOU "IOU1" [c0824fb6-3d31-44b8-bea5-8719fe874952]: assigned with application ID 1 2024-03-30 10:49:57 INFO iou_vm.py:875 IOU "IOU1" [c0824fb6-3d31-44b8-bea5-8719fe874952]: number of Ethernet adapters changed to 2 2024-03-30 10:49:57 INFO iou_vm.py:903 IOU "IOU1" [c0824fb6-3d31-44b8-bea5-8719fe874952]: number of Serial adapters changed to 2 IOS On Unix - Cisco Systems confidential, internal use only 2024-03-30 10:49:57 INFO iou_vm.py:288 IOU "IOU1" [c0824fb6-3d31-44b8-bea5-8719fe874952]: RAM updated from 256MB to 768MB 2024-03-30 10:49:57 INFO iou_vm.py:875 IOU "IOU1" [c0824fb6-3d31-44b8-bea5-8719fe874952]: number of Ethernet adapters changed to 4 2024-03-30 10:49:57 INFO iou_vm.py:145 IOU "IOU1" [c0824fb6-3d31-44b8-bea5-8719fe874952]: IOU image updated to "/home/sysadmin/GNS3/images/IOU/x86_64_crb_linux_l2-adventerprisek9-ms" 2024-03-30 10:49:57 INFO iou_vm.py:903 IOU "IOU1" [c0824fb6-3d31-44b8-bea5-8719fe874952]: number of Serial adapters changed to 0 2024-03-30 10:49:57 INFO web_log.py:211 127.0.0.1 [30/Mar/2024:10:49:57 +0000] "POST /v2/compute/projects/9ac13950-01c8-492b-83ac-fb65fb205c75/iou/nodes HTTP/1.1" 201 899 "-" "Python/3.10 aiohttp/3.9.3" 2024-03-30 10:49:57 INFO web_log.py:211 192.168.1.50 [30/Mar/2024:10:49:57 +0000] "POST /v2/projects/9ac13950-01c8-492b-83ac-fb65fb205c75/open HTTP/1.1" 201 767 "-" "GNS3 QT Client v2.2.46" 2024-03-30 10:49:57 INFO project_handler.py:272 New client has connected to the notification stream for project ID '9ac13950-01c8-492b-83ac-fb65fb205c75' (WebSocket method) 2024-03-30 10:49:57 INFO web_log.py:211 192.168.1.50 [30/Mar/2024:10:49:57 +0000] "GET /v2/projects/9ac13950-01c8-492b-83ac-fb65fb205c75/nodes HTTP/1.1" 200 6922 "-" "GNS3 QT Client v2.2.46" 2024-03-30 10:49:57 INFO web_log.py:211 192.168.1.50 [30/Mar/2024:10:49:57 +0000] "GET /v2/projects/9ac13950-01c8-492b-83ac-fb65fb205c75/links HTTP/1.1" 200 203 "-" "GNS3 QT Client v2.2.46" 2024-03-30 10:49:57 INFO web_log.py:211 192.168.1.50 [30/Mar/2024:10:49:57 +0000] "GET /v2/projects/9ac13950-01c8-492b-83ac-fb65fb205c75/drawings HTTP/1.1" 200 206 "-" "GNS3 QT Client v2.2.46" 2024-03-30 10:50:00 INFO web_log.py:211 192.168.1.50 [30/Mar/2024:10:50:00 +0000] "GET /v2/gns3vm HTTP/1.1" 200 378 "-" "GNS3 QT Client v2.2.46" 2024-03-30 10:50:00 INFO web_log.py:211 192.168.1.50 [30/Mar/2024:10:50:00 +0000] "GET /v2/iou_license HTTP/1.1" 200 277 "-" "GNS3 QT Client v2.2.46" 2024-03-30 10:50:00 INFO init.py:70 Load controller configuration file /home/sysadmin/.config/GNS3/2.2/gns3_controller.conf 2024-03-30 10:50:00 INFO web_log.py:211 192.168.1.50 [30/Mar/2024:10:50:00 +0000] "GET /v2/gns3vm/engines HTTP/1.1" 200 1317 "-" "GNS3 QT Client v2.2.46" 2024-03-30 10:50:00 INFO web_log.py:211 192.168.1.50 [30/Mar/2024:10:50:00 +0000] "GET /v2/gns3vm/engines/vmware/vms HTTP/1.1" 200 203 "-" "GNS3 QT Client v2.2.46" 2024-03-30 10:50:04 INFO init.py:70 Load controller configuration file /home/sysadmin/.config/GNS3/2.2/gns3_controller.conf 2024-03-30 10:50:04 INFO web_log.py:211 192.168.1.50 [30/Mar/2024:10:50:04 +0000] "PUT /v2/iou_license HTTP/1.1" 201 282 "-" "GNS3 QT Client v2.2.46" 2024-03-30 10:50:07 INFO iou_vm.py:414 Checking IOU license in '/tmp/tmpv0dr5s34/iourc' 2024-03-30 10:50:07 INFO base_node.py:701 Starting new uBridge hypervisor 0.0.0.0:34487 2024-03-30 10:50:07 INFO hypervisor.py:169 starting ubridge: ['/usr/bin/ubridge', '-H', '0.0.0.0:34487'] 2024-03-30 10:50:07 INFO hypervisor.py:171 logging to /home/sysadmin/GNS3/projects/9ac13950-01c8-492b-83ac-fb65fb205c75/project-files/iou/c0824fb6-3d31-44b8-bea5-8719fe874952/ubridge.log 2024-03-30 10:50:07 INFO hypervisor.py:179 ubridge started PID=1272 2024-03-30 10:50:07 INFO base_node.py:704 Hypervisor 0.0.0.0:34487 has successfully started 2024-03-30 10:50:07 INFO ubridge_hypervisor.py:83 Connected to uBridge hypervisor on 127.0.0.1:34487 after 0.1007 seconds 2024-03-30 10:50:07 INFO iou_vm.py:780 IOU IOU1 [id=c0824fb6-3d31-44b8-bea5-8719fe874952]: NETMAP file created IOS On Unix - Cisco Systems confidential, internal use only 2024-03-30 10:50:07 INFO iou_vm.py:556 Starting IOU: ['/home/sysadmin/GNS3/images/IOU/x86_64_crb_linux_l2-adventerprisek9-ms', '-e', '4', '-s', '0', '1'] 2024-03-30 10:50:07 INFO iou_vm.py:565 IOU instance c0824fb6-3d31-44b8-bea5-8719fe874952 started PID=1277 2024-03-30 10:50:07 INFO web_log.py:211 127.0.0.1 [30/Mar/2024:10:50:07 +0000] "POST /v2/compute/projects/9ac13950-01c8-492b-83ac-fb65fb205c75/iou/nodes/c0824fb6-3d31-44b8-bea5-8719fe874952/start HTTP/1.1" 200 991 "-" "Python/3.10 aiohttp/3.9.3" 2024-03-30 10:50:07 INFO web_log.py:211 192.168.1.50 [30/Mar/2024:10:50:07 +0000] "POST /v2/projects/9ac13950-01c8-492b-83ac-fb65fb205c75/nodes/c0824fb6-3d31-44b8-bea5-8719fe874952/start HTTP/1.1" 200 6187 "-" "GNS3 QT Client v2.2.46" 2024-03-30 10:50:11 INFO iou_vm.py:1295 saving startup-config to /home/sysadmin/GNS3/projects/9ac13950-01c8-492b-83ac-fb65fb205c75/project-files/iou/c0824fb6-3d31-44b8-bea5-8719fe874952/startup-config.cfg 2024-03-30 10:50:11 INFO web_log.py:211 192.168.1.50 [30/Mar/2024:10:50:11 +0000] "GET /v2/computes HTTP/1.1" 200 1022 "-" "GNS3 QT Client v2.2.46" 2024-03-30 10:50:15 INFO iou_vm.py:1295 saving startup-config to /home/sysadmin/GNS3/projects/9ac13950-01c8-492b-83ac-fb65fb205c75/project-files/iou/c0824fb6-3d31-44b8-bea5-8719fe874952/startup-config.cfg 2024-03-30 10:50:15 INFO web_log.py:211 192.168.1.50 [30/Mar/2024:10:50:15 +0000] "GET /v2/computes HTTP/1.1" 200 1022 "-" "GNS3 QT Client v2.2.46"

grossmj commented 2 months ago

Not sure what happened, return code -6 means that the IOU process failed with SIGABRT. I don't think we can do anything about it.

@Zero3Dash should we close this issue?

Zero3Dash commented 2 months ago

@grossmj I've build 2 VM's on Ubuntu 20.04.04 and 22.04.04 over the weekend.

It runs fine on 22.04.04, but fails with the -6 error code on 20.04.04. Which would indicate its something within the Linux Distro.

If it's not something that will be fix in the here and now, but later when the distro gets upgraded. Then, it might be worth including in the appliance file that the version IOU L2 v17.12.1 will only successfully run on GNS3 running 22.04.04 and possible higher..

I'm happy for you to close it, btw.

TheLordOfTheKings commented 2 months ago

Hey guys, @Zero3Dash @grossmj Please don't close the issue yet, I have the exact problem with the latest L2 image (17.12.1). Can i upgrade my current GNS3 2.2.46 server with ubuntu 20.04 to 22.04 ? how? Thanks.

Zero3Dash commented 2 months ago

@arash88 If you are using the pre-compiled VM, you technically do not.

What I did was to create a new VM using Ubuntu Server 22.04 ISO, then manually installed GNS3 2.2.46, using the following guide. https://docs.gns3.com/docs/getting-started/installation/linux/

Zero3Dash commented 2 months ago

closed

b-ehlers commented 2 months ago

I'm using a GNS3 server running directly on Debian 12 (Bookworm) and I had the same issue. I could resolve it by commenting/deleting the xml.cisco.com entry in my /etc/hosts.

Then I tested the GNS3VM (based on Ubuntu 20.04/focal). Also here the issue was resolved by removing the xml.cisco.com entry in /etc/hosts.

So it is not necessary to switch to a new OS.

grossmj commented 2 months ago

@arash88 please try the suggestion above and let me know if it works for you.

Zero3Dash commented 2 months ago

@grossmj It works for me, not tested the impact on other images.

On and unrelated note, I've also notice in the 17.12.1 base startup config file, "no ip domain-lookup" should be "no ip domain lookup"

b-ehlers commented 2 months ago

It works for me, not tested the impact on other images.

xml.cisco.com is currently not used by cisco, so it's unlikely that it has an impact. And I have noticed no impact on other images.

$ host xml.cisco.com 8.8.8.8
Using domain server:
Name: 8.8.8.8
Address: 8.8.8.8#53
Aliases: 

Host xml.cisco.com not found: 3(NXDOMAIN)
grossmj commented 2 months ago

I can read the content inside but can't make a change. Nano opens empty.

That's strange, I don't have this issue.

Screenshot from 2024-04-06 12-08-30

Maybe try with vi?

TheLordOfTheKings commented 2 months ago

Thanks @grossmj, chmod and chown helped me edit it. Everything works fine now! I can confirm this method works. Also we still can block xml.cisco.com without messing things up by adding this rule to iptables:

iptables -I OUTPUT -p udp --dport 53 -m string --hex-string "|03|xml|05|cisco|03|com" --algo bm -j DROP

grossmj commented 2 months ago

Also we still can block xml.cisco.com without messing things up by adding this rule to iptables.

I have created this PR https://github.com/GNS3/gns3-server/pull/2366

b-ehlers commented 2 months ago

@grossmj What about removing xml.cisco.com from /etc/hosts in the gns3-vm? Currently it is set in scripts/provision.sh in every branch of the gns3-vm.

scripts/provision.sh

# Block IOU phone home call
echo "127.0.0.254 xml.cisco.com" | sudo tee --append /etc/hosts
grossmj commented 2 months ago

@grossmj What about removing xml.cisco.com from /etc/hosts in the gns3-vm? Currently it is set in scripts/provision.sh in every branch of the gns3-vm.

I am considering it. I haven't removed it yet because the problem only affects Ubuntu Focal, so only older GNS3 VMs. For future releases, should we block xml.cisco.com using iptables instead or totally remove any blocking?

b-ehlers commented 2 months ago

I haven't removed it yet because the problem only affects Ubuntu Focal, so only older GNS3 VMs.

Just tested the GNSVM image GNS3.VM.KVM.2.2.46.zip, it uses focal. I don't know, what OS the other GNS3 VMs are using, but for me it looks, that focal is still actively distributed as the GNS3 VM base OS.

For future releases, should we block xml.cisco.com using iptables instead or totally remove any blocking?

I'm not sure. The iputils command blocks the DNS request to xml.cisco.com, so that a DNS lookup to that address will time out. That may slow down the IOU startup, but I haven't noticed a delay. If jammy doesn't show the issue, I won't add the iputils command. The xml.cisco.com entry in /etc/hosts should be sufficient.

grossmj commented 2 months ago

Just tested the GNSVM image GNS3.VM.KVM.2.2.46.zip, it uses focal. I don't know, what OS the other GNS3 VMs are using, but for me it looks, that focal is still actively distributed as the GNS3 VM base OS.

That's right, I forgot 2.2.x still uses Ubuntu Focal and 3.x uses Jammy.

I think I will just remove xml.cisco.com from the hosts file and not mess with iptables.