osrf / subt

This repostory contains software for the virtual track of the DARPA SubT Challenge. Within this repository you will find Gazebo simulation assets, ROS interfaces, support scripts and plugins, and documentation needed to compete in the SubT Virtual Challenge.
Other
309 stars 97 forks source link

Ignition crashing #80

Closed osrf-migration closed 5 years ago

osrf-migration commented 5 years ago

Original report (archived issue) by Jon Fink (Bitbucket: jonfink-arl).

The original report had attachments: error_log.txt


Running into issues where Ignition crashes with a segfault and hangs (requiring many processes to be brought down by hand). Happens under a variety of circumstances, in this case with the addition of a 3D Grid to the main window (after ign launch -v 4 virtual_stix.ign robotName1:=X1 robotConfig1:=X1_SENSOR_CONFIG_4 hangs during start, will create another issue for that).

Relevant output includes:

[GUI] [Dbg] [Application.cc:305] Loading plugin [Grid3D]                                                                                 
[GUI] [Msg] Added plugin [3D Grid] to main window                                                                                        [GUI] [Msg] Loaded plugin [Grid3D] from path [/usr/lib/x86_64-linux-gnu/ign-gui-2/plugins/libGrid3D.so]                                  
[GUI] [Msg] Loading plugin [ignition-rendering2-ogre]                                                                                    [GUI] [Wrn] [Grid3D.cc:214] Scene "scene" not found, Grid plugin won't work.                                                             
/usr/lib/ruby/ignition/cmdlaunch1.rb:202: [BUG] Segmentation fault at 0x0000000000000001                                                 ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-linux-gnu]                                                                             
                                                                                                                                         -- Control frame information -----------------------------------------------                                                             
c:0005 p:---- s:0044 e:000043 CFUNC  :call                                                                                               c:0004 p:0017 s:0039 e:000038 METHOD /usr/lib/ruby/ignition/cmdlaunch1.rb:202                                                            
c:0003 p:0446 s:0033 e:000032 METHOD /usr/lib/ruby/ignition/cmdlaunch1.rb:202                                                            c:0002 p:0628 s:0019 E:000688 EVAL   /usr/bin/ign:267 [FINISH]                                                                           
c:0001 p:0000 s:0003 E:002380 (none) [FINISH]                                                                                                                                                                                                                                     
-- Ruby level backtrace information ----------------------------------------                                                             /usr/bin/ign:267:in `<main>'                                                                                                             
/usr/lib/ruby/ignition/cmdlaunch1.rb:202:in `execute'                                                                                    /usr/lib/ruby/ignition/cmdlaunch1.rb:202:in `run'                                                                                        
/usr/lib/ruby/ignition/cmdlaunch1.rb:202:in `call'                                                                                                                                                                                                                                
-- Machine register context ------------------------------------------------                                                              RIP: 0x00007f0186e1e6df RBP: 0x00007f0187ffe820 RSP: 0x00007f0187ffe6b0                                                                 
 RAX: 0x00007f0187445c68 RBX: 0x00007f01803f7950 RCX: 0x00007f01877b5398                                                                  RDX: 0x0000000000000000 RDI: 0x00007f01803f7950 RSI: 0x0000000000000001                                                                 
  R8: 0x00007f01802eaf90  R9: 0x00007f01877b5398 R10: 0x0000000000000048                                                                  R11: 0x00007f0160be9070 R12: 0x00007f01803f7950 R13: 0x00007f0187ffe9b0                                                                 
 R14: 0x00007f0187ffeac0 R15: 0x000055906b27e754 EFL: 0x0000000000010202                                                                                                                                                                                                          
-- C level backtrace information -------------------------------------------                                                             /usr/lib/x86_64-linux-gnu/libruby-2.5.so.2.5(0x7f01d1893965) [0x7f01d1893965]                                                            
/usr/lib/x86_64-linux-gnu/libruby-2.5.so.2.5(0x7f01d1893b9c) [0x7f01d1893b9c]                                                            /usr/lib/x86_64-linux-gnu/libruby-2.5.so.2.5(0x7f01d175d884) [0x7f01d175d884]                                                            
/usr/lib/x86_64-linux-gnu/libruby-2.5.so.2.5(0x7f01d18239a2) [0x7f01d18239a2]                                                            /lib/x86_64-linux-gnu/libc.so.6(0x7f01d1323f20) [0x7f01d1323f20]                                                                         
/usr/lib/x86_64-linux-gnu/libOgreMain.so.2.1.0(_ZN4Ogre7Frustum16enableReflectionERKNS_5PlaneE+0xf) [0x7f0186e1e6df]                     /usr/lib/x86_64-linux-gnu/libOgreMain.so.1.9.0(_ZNK4Ogre7Frustum13getViewMatrixEv+0xd) [0x7f0160c3593d]                                  
/usr/lib/x86_64-linux-gnu/libOgreMain.so.1.9.0(_ZNK4Ogre6Camera22getCameraToViewportRayEffPNS_3RayE+0x3a) [0x7f0160be897a]               /usr/lib/x86_64-linux-gnu/libOgreMain.so.1.9.0(_ZNK4Ogre6Camera22getCameraToViewportRayEff+0x36) [0x7f0160be90a6]                        
/usr/lib/x86_64-linux-gnu/libignition-rendering2-ogre2.so(_ZN8ignition9rendering2v213Ogre2RayQuery13SetFromCameraERKSt10shared_ptrINS1_6CameraEERKNS_4math2v67Vector2IdEE+0xc4) [0x7f0187577e64]                                                                                  
/usr/lib/x86_64-linux-gnu/ign-gazebo-2/plugins/gui/libGzScene3D.so(_ZNK8ignition6gazebo2v211IgnRenderer13ScreenToSceneERKNS_4math2v67Vector2IiEE+0xdc) [0x7f0191dee2fc]                                                                                                           
/usr/lib/x86_64-linux-gnu/ign-gazebo-2/plugins/gui/libGzScene3D.so(_ZN8ignition6gazebo2v211IgnRenderer22HandleMouseViewControlEv+0x3c8) [0x7f0191df1c38]                                                                                                                          
/usr/lib/x86_64-linux-gnu/ign-gazebo-2/plugins/gui/libGzScene3D.so(_ZN8ignition6gazebo2v211IgnRenderer16HandleMouseEventEv+0x3a) [0x7f0191df1cfa]                                                                                                                                 
/usr/lib/x86_64-linux-gnu/ign-gazebo-2/plugins/gui/libGzScene3D.so(_ZN8ignition6gazebo2v211IgnRenderer6RenderEv+0xd7) [0x7f0191df1e07]   /usr/lib/x86_64-linux-gnu/ign-gazebo-2/plugins/gui/libGzScene3D.so(_ZN8ignition6gazebo2v212RenderThread10RenderNextEv+0x46) [0x7f0191df21
a6]                                                                                                                                      /usr/lib/x86_64-linux-gnu/libQt5Core.so.5(_ZN7QObject5eventEP6QEvent+0xe2) [0x7f01cb7ba122]                                              
/usr/lib/x86_64-linux-gnu/libQt5Core.so.5(_ZN16QCoreApplication6notifyEP7QObjectP6QEvent+0x5a) [0x7f01cb78a83a]                          
/usr/lib/x86_64-linux-gnu/libQt5Core.so.5(_ZN16QCoreApplication15notifyInternal2EP7QObjectP6QEvent+0x118) [0x7f01cb78a9a8]               
/usr/lib/x86_64-linux-gnu/libQt5Core.so.5(_ZN23QCoreApplicationPrivate16sendPostedEventsEP7QObjectiP11QThreadData+0x1ed) [0x7f01cb78d11d]
/usr/lib/x86_64-linux-gnu/libQt5Core.so.5(0x7f01cb7e42c3) [0x7f01cb7e42c3]                                                               
/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0(g_main_context_dispatch+0x2e7) [0x7f01c83fb317]                                               
/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0(0x7f01c83fb550) [0x7f01c83fb550]                                                              
/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0(g_main_context_iteration+0x2c) [0x7f01c83fb5dc]                                               
/usr/lib/x86_64-linux-gnu/libQt5Core.so.5(_ZN20QEventDispatcherGlib13processEventsE6QFlagsIN10QEventLoop17ProcessEventsFlagEE+0x5f) [0x7f
01cb7e38ef]                                                                                                                              
/usr/lib/x86_64-linux-gnu/libQt5Core.so.5(_ZN10QEventLoop4execE6QFlagsINS_17ProcessEventsFlagEE+0x13a) [0x7f01cb7889ea]                  
/usr/lib/x86_64-linux-gnu/libQt5Core.so.5(_ZN7QThread4execEv+0x6a) [0x7f01cb5a722a]                                                      
/usr/lib/x86_64-linux-gnu/libQt5Core.so.5(0x7f01cb5ac16d) [0x7f01cb5ac16d]                                                               
/lib/x86_64-linux-gnu/libpthread.so.0(start_thread+0xdb) [0x7f01d10cd6db]                                                                
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f01d140688f] ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

At this point the command hangs at the terminal and does not respond to Ctrl+C. Killing the ign process with -9 leaves a number of processes still running that need to be brought down by hand.

osrf-migration commented 5 years ago

Original comment by Jon Fink (Bitbucket: jonfink-arl).


Some additional (related?) information is that even when ignition comes down correctly, it seems to leave the two roslaunch commands that it begins running.

osrf-migration commented 5 years ago

Original comment by Nate Koenig (Bitbucket: Nathan Koenig).


@chapulina , I think the stuck roslaunch commands are the ros1 bridge. Do you happen to have any ideas there?

osrf-migration commented 5 years ago

Original comment by Nate Koenig (Bitbucket: Nathan Koenig).


osrf-migration commented 5 years ago

Original comment by Louise Poubel (Bitbucket: chapulina, GitHub: chapulina).


addition of a 3D Grid to the main window

Is that from the menu on the top-right? That has never worked, we should remove it from the menu.

leave the two roslaunch commands that it begins running

Do you know exactly which ones? I see 7 roslaunch calls inside virtual_stix.ign

osrf-migration commented 5 years ago

Original comment by Louise Poubel (Bitbucket: chapulina, GitHub: chapulina).


even when ignition comes down correctly, it seems to leave the two roslaunch commands that it begins running

Also, what do you mean by "comes down correctly"? When I Ctrl + C the terminal, all processed are finished correctly. If I close the window, other processes stay alive though, but I wouldn't expect them to be brought down.

osrf-migration commented 5 years ago

Original comment by Jon Fink (Bitbucket: jonfink-arl).


This is only happening when some part of ignition crashes, e.g., when adding the 3D grid via the menu in the top-right. I have experienced other times where the ign launch ... command never finishes bringing up the simulator (I get the GUI window popping up but do not see the 3D scene). Ctrl+C at the terminal does not work properly at this point resulting in my issues above.

In short, it seems there may be something blocking the correct shutdown of executables that have been launched via ignition if one part of the ignition system crashes.

osrf-migration commented 5 years ago

Original comment by Louise Poubel (Bitbucket: chapulina, GitHub: chapulina).


Thanks for the clarification. It would be helpful to have a way to reproduce the issue. Even if I crash ign-gazebo by inserting Grid3D, I'm being able to shutdown the other processes with Ctrl+C.

osrf-migration commented 5 years ago

Original comment by Jon Fink (Bitbucket: jonfink-arl).


I’m still try to figure out a reliable way to reproduce - definitely intermittent.

osrf-migration commented 5 years ago

Original comment by Jon Fink (Bitbucket: jonfink-arl).


osrf-migration commented 5 years ago

Original comment by baila (Bitbucket: baila).


Hey Jon Fink (jonfink-arl) , I guess I am also running into something similar. I have added my comment with logs and screenshot at [https://osrf-migration.github.io/subt-gh-pages/#!/osrf/subt/issues/82/problems-with-examplesetup-tutorial (#82)](https://osrf-migration.github.io/subt-gh-pages/#!/osrf/subt/issues/82/problems-with-examplesetup-tutorial (#82))

Difference is that, I used competition.launch file in y experiment.

Please share if you have found any workaround.

osrf-migration commented 5 years ago

Original comment by baila (Bitbucket: baila).


Hi Nate Koenig (Nathan Koenig) , I am adding one file with all the logs during the run of the cmd :

ign launch virtual_stix.ign robotName1:=X1 robotConfig1:=X1_SENSOR_CONFIG_1

I get the window with the map only but no robot spawned.

The issue should be fixed very soon…I am stuck completely. Can’t work after moving to ignition framework. :disappointed:

osrf-migration commented 5 years ago

Original comment by baila (Bitbucket: baila).


osrf-migration commented 5 years ago

Original comment by Nate Koenig (Bitbucket: Nathan Koenig).


I think you need to update your ros1_ign_bridge code. Are you building ros1_ign_bridge from source or using debians?

osrf-migration commented 5 years ago

Original comment by Nate Koenig (Bitbucket: Nathan Koenig).


Did this fix your problem?

osrf-migration commented 5 years ago

Original comment by Alfredo Bencomo (Bitbucket: bencomo).


Reopen if it still crashes with latest ign packages and tunnel_circuit branch.

osrf-migration commented 5 years ago

Original comment by Alfredo Bencomo (Bitbucket: bencomo).