How do I troubleshoot a node that does not accept console connections?

[1001QAdotNET]

From time to time some of my nodes do not accept console connections anymore. The only solution I was able to find is to restart the node which is time consuming if you use some of the newer Cisco appliances that require a lot of memory and CPU

If you try to connect from an adjacent node in the topology that has IP connectivity to that node that works ...so the node is not dead is it just the console connection that is gone

[cristian-ciobanu]

I can confirm I experience this issue sometimes. Mostly when I leave the topology running for a long time. So far restart only seems to work.

[1001QAdotNET]

Yes I should have mentioned that, it if for long time running instances or projects

[grossmj]

This should have been fixed with version 2.2. Please comment back if you see the same issue in version 2.2.x or later. Thanks.

[1001QAdotNET]

You might be right on this I am on 2.2.0x now not sure what x is I will look at it over the weekend

[grossmj]

I meant version 2.2 or later. Thanks.

[1001QAdotNET]

no this is still a problem, Client version 2.2.0b4, server (controller) version 2.2.0dev15 I left a 24 IOSv topology running over night ...the consoles are all dead and I can't reach them anymore

[grossmj]

Can you reproduce this with any Qemu node? How long do you have to wait for it to happen?

[1001QAdotNET]

IOSv is a qemu node. Not sure how long, I left the node running and accessible at 2AM ...at 11AM the problem was there

[grossmj]

@1001QAdotNET is there a chance your system went to sleep mode?

[1001QAdotNET]

Are you referring to the client ? Yes I put the client to sleep more over night The server is an ESXi6.5 and I wish I could put it to sleep, that feature is lost n 6.5 (standalone)

The project is set to keep running after the client disconnects

[cristian-ciobanu]

In my case I use the Leave the project running on the server when I close the GUI. When I open again the project next day for example some of the nodes (Qemu appliances) refuse to connect to the console. I experienced this only with nodes which use telnet as console not VNC or Spice.

[1001QAdotNET]

yes that is the same for me

[b-abadie]

Hello, I have the same issue on gns3-server 2.1.21~bionic1 (on an Ubuntu server) and gns3-gui 2.1.21. Project is configured to keep running in background when closing the GNS3 GUI. I run into the issue when re-connecting in the morning. When clocking off for the day, I usually just put my laptop into sleep mode without closing my VPN or my telnet sessions, so there might be some timeouts going on server side. In term of timing/recurrence, having to reload a node (all Qemu Arista vEOS by the way) in the morning is the norm, and having my telnet session still functional is the exception. But I never encounter any console dropout during the day so I'd say it happens after 8h at least. Also note that, when trying to re-open a console session without reloading the node, the telnet connects fine (i.e. the socket is open on server side) but there is no intput/output so there might be a mapping issue inside gns3-server.

I would be happy to help you troubleshoot this, here or via IM (french/english)

[grossmj]

Another report: https://gns3.com/community/discussion/securecrt-loses-console-access

[b-abadie]

I had the issue again this morning, and tried tcpdumping the telnet connection between gns3server and qemu-system-x86 on the server's loopback (tcpdump -i lo port 5001 -XX -s 0) while hitting some keys on the console window (i.e. the "public" telnet connection between my telnet client and gns3server). I can confirm packets to and from the VM are sent and received. From the hex dump I can even figure out the keypresses and the switch's hostname and prompt.

When I dump the public telnet connection, I only get ACKs from gns3server.

[spikefishjohn]

Is this still https://github.com/GNS3/gns3-server/issues/1335?

I was looking at https://github.com/GNS3/gns3-server/commit/bb284d082a89fc812a01250cfa3a7f154c5dc1dc

And couldn't figure something out. I see where you make the socket with keep alive and no delay but I don't understand where the socket object got used. Granted major python novice so maybe i'm missing something obvious.

BTW i'm running 2.2.3 and I see the same thing where IOSv console just goes away on long lived connections.

[grossmj]

More reports: https://gns3.com/community/discussion/console-is-not-responding-anymor https://gns3.com/community/discussion/vmx-cli-freezes-every-few-hours-

Looks like it happens only when the GNS3 server is remote.

[1001QAdotNET]

yes indeed mine runs in a VM hosted by an ESXi server

[rafi-d]

Same issue here (2.2.11) . Running on localhost/loopback. After a few minutes, a new VPC just refuses to connect to a console any more . (Latest Win 10 ) .

[grossmj]

The console to node bridge implementation is going to be refactored in a future version of GNS3. In the meantime, I have added a method to reset the console for all nodes (Accessible from GUI menu: Control -> Reset all console connections).

[mikhi-nde]

The console to node bridge implementation is going to be refactored in a future version of GNS3. In the meantime, I have added a method to reset the console for all nodes (Accessible from GUI menu: Control -> Reset all console connections).

Hi, How can I get this update?

[grossmj]

Hi, How can I get this update?

You can download nightly build on https://sourceforge.net/projects/gns-3/files/Nightly%20Builds/

The updated in on 2.3 branch however it is currently broken for macOS: https://github.com/GNS3/gns3-gui/issues/3020#issuecomment-665536001

[competron]

@grossmj i try "Reset all console connections" but after wait several minutes, erorr appear with message "Timeout error for POST call to http://127.0.0.1:3080/v2/compute/projects/fd43ef70-433e-4952-b616-76a8123db2dd/qemu/nodes/04e139d6-09e3-4d44-bfb2-ef3574fc65bb/console/reset after 240s"

My GNS3 VM is hosting in ESXi

[SudeepGoswami]

@grossmj I cannot locate the 2.3 branch in the nightly build section https://sourceforge.net/projects/gns-3/files/Nightly%20Builds/. Can you please point me to the right location? Still looking for the 'console hang' fix. Please help. Thx.

[competron]

Is there any temporary solution for this?

[DN0000]

Is it feasible for 831ee5f46831c669f2e73cd55894d1426f60ef66 to be backported to 2.2.X?

[cristian-ciobanu]

Is it feasible for 831ee5f to be backported to 2.2.X?

I definitely agree these two commits https://github.com/GNS3/gns3-gui/commit/6a9440c978a03ad5c552d5c97d6306f3906986a6 and https://github.com/GNS3/gns3-server/commit/831ee5f46831c669f2e73cd55894d1426f60ef66 which allows to reset all console connections should be ported to release 2.2.x.

I do not see this commit as a breaking change for the 2.2.x branch since it only adds an option to reset the console connections.

I experienced this issue many times when I use some large topologies which are left running in the background after closing the GUI or when I lose connectivity to the GNS3 server and the try to reconnect with the client. In my case I need to stop around 40 - 50 running devices and start all them again in order to be able to gain console access using telnet.

[competron]

@grossmj

I think this feature still not working, I host my GNS3VM in ESXi and tried to reset all console but getting an error. after that, all of the node can't console anymore and need to restart

[cristian-ciobanu]

I also confirm that the Reset console connections feature does not work. When used the console connections of all nodes freeze and they only way is to restart all of them.

[jacquen]

I confirm this problem still exist on version 2.2.34, reset console didn't work. I can get 1 or 2 devices telenet console after 1 hour not touch the lab and put project running at background, but rest of 4-5 nodes just hanging. So I must reset whole lab.

[grossmj]

My last commit should have fixed the problem. Let's test/confirm after v2.2.36 is released.

[cristian-ciobanu]

My last commit should have fixed the problem. Let's test/confirm after v2.2.36 is released.

Just upgraded to the latest 2.2.36 and indeed the Reset console connection feature works fine now. Thanks for fixing this annoying issue One small note: In the GUI the option Control --> Reset all console connections menu item is missing an icon.

[sliddjur]

Why does this keep happening? Even on 2.2.42. I use gns3vm, and even using telnet localhost 5053 doesnt work. I have been reloading all my nodes, until today when I found the Control --> Reset all console connections option, which only works sometimes. I have an IOU image which doesnt reset console properly. i86bi-linux-l3-adventerprisek9-15.4.1T.bin is the image name. Seems to work on QEMU images though.

[grossmj]

@sliddjur there is a sneaky bug in our telnet bridge to node consoles which has never be found, we have plans to rewrite all of it using telnetlib3 (the server implementation wasn't available in this library when we created our Telnet bridge years ago).

[imtssir]

the same issue on 2.2.44.1 confirmed,4 or maybe 5 hours later,when i hit the console button to log in ,it do not accept console connections any more .Control --> Reset all console connections doesn't work . qemu image :CiscoIOSvL215.2(20200924:215240)