search

GNS3 / gns3-server

GNS3 server
GNU General Public License v3.0
806 stars 263 forks source link

[2.1.0dev6] Fedora: Dynamips stops during router boot #1177

Closed AJNOURI closed 6 years ago

AJNOURI commented 7 years ago

https://youtu.be/CzC479gOKwY

grossmj commented 6 years ago

@AJNOURI what version of Dynamips have you?

grossmj commented 6 years ago

Another report: https://gns3.com/qa/dynamips-hypervisor-process-ha-2

sallyhaj commented 6 years ago

I am using Fedora 27 with gns3-2.1.0 but I get a problem when I run any router Dynamips hypervisor process has stopped, return code: -11 C3600 'Ghost': generic_nvram_extract_config: Unknown error -61605

grossmj commented 6 years ago

@jalalsfs how did you install Dynamips, did you compile it?

sallyhaj commented 6 years ago

@grossmj Yes, I have compiled it in Fedora 27 which has gcc (GCC) 7.2.1 compiler. I've just finished trying debian's Dynamips, I replaced the one in /usr/loca/bin with it, then it asks for a libpcap.so.0.8 library, I made a link of libpcap.so.1.8.1, but unfortunately the same result


=> Dynamips hypervisor process has stopped, return code: -11
C2691 'Ghost': generic_nvram_extract_config: Unknown error -61605

then If I try once again I get this message

Timeout after 30 seconds for request http://admin@127.0.0.1:3080/v2/computes
Error while getting compute list: Operation timeout
Aborting request for http://admin@127.0.0.1:3080/v2/settings
Timeout after 60 seconds for request http://admin@127.0.0.1:3080/v2/projects/58cf0c6d-667e-47d2-bed3-7ae4c8ca6889/nodes/c213e7ec-8bae-4780-9164-e1cb3a51ed6d

In that time it just repeating the message again and again, I have then to kill the process.

omadjoudj commented 6 years ago

Still cannot reproduce this issue with the folowing images.

I even rebuilt dynamips just to check if it's not related to new gcc version [1] (dnf clean all ; dnf update if you have gns3-extra copr repo enabled). [1] https://copr.fedorainfracloud.org/coprs/athmane/gns3-extra/build/683460/

@jalalsfs could you check if you have anything related to dynamips crash abrt-cli ls journalctl -a --no-pager | grep -E 'gns3|dynamips'

sallyhaj commented 6 years ago

@athmane Yes, I have update dynamips from gns3-extra, but the problem is the same

=> Dynamips hypervisor process has stopped, return code: -11
C7200 'Ghost': generic_nvram_extract_config: Unknown error -61605

Dynamips hypervisor process has stopped, return code: -11
C3745 'Ghost': generic_nvram_extract_config: Unknown error -61605

and why you don't update it to 0.2.17? Do you mean some IOS are working fine with you and some not?

there is nothing with abrt-cli command related to gns3 or dynamips, and the second one, it's a very long lines in https://paste.fedoraproject.org/paste/KwefS7HX-G0xtvOIHK8YlA

I have just installed CentOS7 only for trying gns3, and I installed it from the source, and it works. I followed most in this file Install GNS3 2.1.0.pdf

omadjoudj commented 6 years ago

From journal, we can see that dynamips has crashed, not sure why abrt didn't catch it. I'm guessing something related to the IOS image that's causing the crash.

Can you check if you have coredumps in /var/lib/systemd/coredump, if not, try to enable core-dumping and reproduce the bug

Dec 03 08:35:34 jalalsfs.org kernel: traps: dynamips[5237] general protection ip:7f6d8218a916 sp:7f6d7a9b2128 error:0 in libc-2.26.so[7f6d82127000+1db000]
Dec 03 08:35:35 jalalsfs.org audit[5219]: ANOM_ABEND auid=1000 uid=1000 gid=1000 ses=2 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 pid=5219 comm="dynamips" exe="/usr/bin/dynamips" sig=11 res=1
Dec 03 08:35:35 jalalsfs.org kernel: traps: dynamips[5244] general protection ip:7f69d436a916 sp:7f69ccf9c128 error:0 in libc-2.26.so[7f69d4307000+1db000]
Dec 03 08:35:44 jalalsfs.org systemd-coredump[5247]: Process 5219 (dynamips) of user 1000 dumped core.
                                                     #6  0x0000560f42187f1d m_flog.part.0 (dynamips)
                                                     #7  0x0000560f421953b2 vm_log (dynamips)
                                                     #8  0x0000560f421ba0ab dev_remote_control_access (dynamips)
                                                     #9  0x0000560f4219e240 mips64_mts32_sw (dynamips)
Dec 03 08:35:45 jalalsfs.org systemd-coredump[5242]: Process 5204 (dynamips) of user 1000 dumped core.
                                                     #6  0x000055ee8b264f1d m_flog.part.0 (dynamips)
                                                     #7  0x000055ee8b2723b2 vm_log (dynamips)
                                                     #8  0x000055ee8b2970ab dev_remote_control_access (dynamips)
                                                     #9  0x000055ee8b27b240 mips64_mts32_sw (dynamips)
sallyhaj commented 6 years ago

Yes, I have in /var/lib/systemd/coredump/ these files

core.dynamips.1000.1c23de78b8ce4646bddcc2a665b01a40.13175.1512332146000000.lz4
core.dynamips.1000.1c23de78b8ce4646bddcc2a665b01a40.13190.1512332169000000.lz4
core.dynamips.1000.1c23de78b8ce4646bddcc2a665b01a40.13388.1512333116000000.lz4
core.dynamips.1000.1c23de78b8ce4646bddcc2a665b01a40.13425.1512333174000000.lz4
core.dynamips.1000.1c23de78b8ce4646bddcc2a665b01a40.13505.1512333920000000.lz4
core.dynamips.1000.88698517755d4e31bb34a1b9708ec992.1003.1512362170000000.lz4
core.dynamips.1000.88698517755d4e31bb34a1b9708ec992.16702.1512366469000000.lz4
core.dynamips.1000.88698517755d4e31bb34a1b9708ec992.16734.1512366489000000.lz4
core.dynamips.1000.88698517755d4e31bb34a1b9708ec992.18636.1512367128000000.lz4
core.dynamips.1000.88698517755d4e31bb34a1b9708ec992.18746.1512369243000000.lz4
core.dynamips.1000.88698517755d4e31bb34a1b9708ec992.988.1512362175000000.lz4
core.gns3.1000.88698517755d4e31bb34a1b9708ec992.16684.1512366591000000.lz4

but I don't know how to read them, Yesterday, I had a fresh F27, and the result

WARNING http_client:612 Timeout after 30 seconds for request http://admin@127.0.0.1:3080/v2/computes
ERROR compute_manager:75 Error while getting compute list: Operation timeout
WARNING http_client:612 Timeout after 60 seconds for request http://admin@127.0.0.1:3080/v2/projects/d7304a33-048c-4c91-bfac-650254aae2bc/nodes/11ede30f-92bc-44f5-9037-a51404f00462
CRITICAL graphics_view.py:1515 Operation timeout
WARNING http_client:612 Timeout after 60 seconds for request http://admin@127.0.0.1:3080/v2/projects/d7304a33-048c-4c91-bfac-650254aae2bc/nodes/f61048ae-1283-4183-a2e6-2e02ffe7078c
CRITICAL graphics_view.py:1515 Operation timeout

I think it's not a Dynamips issue, it maybe something related to it or something Dynamips depends on. Dynampis, I tried the one you packaged, I tried the one from debian which is working on debian just fine, I tried one I've done compiled it on CentOS and it is working okay with CentOS but when I copy it to Fedora it is not working.

IOS, I am using the same IOSs on all distros, I have only 4-5 IOS, so, I don't think it's IOS fault, because they are working with debian and centos, not with fedora.

omadjoudj commented 6 years ago

@jalalsfs could you please send me most recent dynamips coredump. (we dont have Bugzilla component for dynamips since it's not packaged in Fedora base repos, that's why abrt cannot catch it)

ddragic commented 6 years ago

@jalalsfs You should be able to use coredumpctl list to list the coredumps, coredumpctl info <PID> for more information and coredumpctl gdb <PID> to run GDB on the coredump. In gdb you can run bt full to get a backtrace and paste it here.

To get useful information out of GDB you should compile dynamips with -DCMAKE_BUILD_TYPE=Debug.

@athmane I can crash dynamips with any IOS image or platform on Fedora 26 64-bit GNS3/dynamips#83. It's segfault-ing in malloc so there's probably some corruption happening earlier.

sallyhaj commented 6 years ago

@ddragic Okay, well, I have just tried coredumpctl gdb <PID> but It asked to download more debug packages dnf debuginfo-install dynamips-0.2.16-1.fc27.x86_64 elfutils-libelf-0.170-1.fc27.x86_64 glibc-2.26-16.fc27.x86_64 libpcap-1.8.1-6.fc27.x86_64 zlib-1.2.11-4.fc27.x86_64 I just tried this with gns3 rpm packages, I will try it once again with compiling dynamips from source. Then after, I couldn't understand the result, please have a look at it in https://pastebin.com/wRdYjZvx

ddragic commented 6 years ago

Looks like athmane's repo has a debuginfo package which you installed so there's no need to build from source.

You didn't print out all of bt full output, press "enter" when gdb ask to continue or use set pagination off before the commands. Also you can attach files with your comment, no need to use a paste bin.

From what I can see your crash does look somewhat similar to the issue I'm having, but getting to the bottom of this requires deeper knowledge of the dynamips vm, I'll have a look at it when I get a chance.

For a workaround you could try installing a 32-bit dynamips build from athmane or building a 32-bit build from source with -DDYNAMIPS_ARCH=x86 and see if it works for you.

sallyhaj commented 6 years ago

@ddragic @athmane Yes, athmanes' repo has it, sorry I didn't know that there is something else after the first page in bt full, I attached two files of different coredumps, because in once around 3 files are created. I have tried 32-bit as you suggested, but I had to remove dynamips and vpcs 64-bit packages first, then I installed 32-bit packages, and GNS3 is WORKING fine. The options -DDYNAMIPS_ARCH=x86 where and when shall I put it? is it after cmake command? coredump1.txt coredump2.txt

ddragic commented 6 years ago

Yes, cmake -DDYNAMIPS_ARCH=x86, but the end result should be the same as 32-bit rpm you installed.

sallyhaj commented 6 years ago

@grossmj @noplay Can we say that, the issue with Fedora gcc 64-bit compiler?

ddragic commented 6 years ago

I would say an issue with glibc.

sallyhaj commented 6 years ago

@ddragic If it's a glibc issue, why gns3 doesn't work with Debian's Dynamips file? Even though it's working perfectly with Debian, I tried today compiling Dynamips with a new update of glibc-2.26-20.fc27.x86_64, but same problem.

grossmj commented 6 years ago

Maybe the problem comes from version 0.2.17. Have you tried with version 0.2.16? https://github.com/GNS3/dynamips/releases/tag/v0.2.16

sallyhaj commented 6 years ago

@grossmj I have just finished the experiment, firstly I installed 0.2.16 as RPM for 64-bit, but it didn't work. Secondly, I installed 0.2.16 as RPM for 32-bit and it's working. Thirdly, I used the source version with 0.2.16, after compiling it it's not working, because my system is 64-bit so by default the compiler .... . and the problem with 64-bit. I think it's an issue with glibc as @ddragic suggested. Note: RPM files from @athmane repo

grossmj commented 6 years ago

I've installed a brand new Fedora 27 running in a VMware VM.

Compiled Dynamips from the sources:

dnf install git cmake gcc elfutils-libelf-devel libpcap-devel
git clone git://github.com/GNS3/dynamips.git
cd dynamips
mkdir build
cd build
cmake ..
make 
make install

[root@localhost build]# dynamips -v
Cisco Router Simulation Platform (version 0.2.17-amd64/Linux stable)
Copyright (c) 2005-2011 Christophe Fillot.
Build date: Jan 30 2018 18:48:39
...

Installed GNS3 from Pypi:

dnf install redhat-rpm-config python3-devel python3-qt5
pip3 install gns3-gui==2.1.3
pip3 install gns3-server==2.1.3

I could run multiple IOS routers (c3745-adventerprisek9-mz.124-25d.bin image) without any issue.

So I think the problem is not with Dynamips or GNS3 too, most likely with glibc as @ddragic suggested. I don't think we can do much on our side, therefore I will close this issue soon.

sallyhaj commented 6 years ago

@grossmj Even me, I don't have any issue with 32-bit Fedora 27, the problem is just with 64-bit. So, have you used 32-bit or 64-bit?

grossmj commented 6 years ago

@jalalsfs I have used 64-bit.