Closed hershwg closed 11 years ago
[jfaust] Hmm -- is this reproducible 100% of the time for you? I've just tried it ~20 times or so with the latest cturtle debs with no luck.
The valgrind problems appear to be roscpp bugs, which may be causing the crash... those I can reproduce by kill -9ing the state publisher, and I can reproduce with just a talker/listener.
[mrinal] Yes, it's reproducible 100% of the time when running without gdb or valgrind. I've been doing this all day since I'm debugging a robot urdf model, so I have to keep restarting the robot_state_publisher. Under gdb or valgrind however, I can never get it to crash at all.
Maybe this bug should be reassigned to roscpp to fix the TransportTCP problems?
[jfaust] Depends if that's the actual cause of the crash. I've opened a separate ticket rather than moving this one: #ros3047
If the fix for that ends up fixing your crash (once I have a fix anyway), we can close this one.
[jfaust] Ok, after tracking it down I doubt the it will fix any crash, but can you try this diff for roscpp? https://code.ros.org/trac/ros/changeset/11356?format=diff&new=11356
[mrinal] Give me a moment, I'll have to install ros from source to test this. Oh, how the debs have spoilt us!
[mrinal] No luck, I still get the crash. I also still get the valgrind messages when I kill -9 though. All I did was get ros from svn (only core ros, not the rest), apply the patch, compile, change ROS_ROOT to the new ros and also add that to the front of my ROS_PACKAGE_PATH, and re-run everything. Shouldn't that be enough?
[jfaust] No, because of the use of rpath it's still going to be using the version of roscpp it was built against. You need to get the stacks through visualization and rebuild.
[rusu] I got the same problem with PointCloud2 visualizers. To reproduce, try this:
{{{
$ roscore
$ rosrun rviz rviz
$ roslaunch
where
}}}
Then, Ctrl+C the roslaunch, and rviz will die. (or does on my machine - CTurtle deb install)
[jfaust] what's your rviz configuration?
[jfaust] well, subscribing with a PointCloud2 display to scene_pointcloud2 doesn't produce the crash for me with the latest cturtle debs. Kevin seems to be having a similar problem: #4463, which points to a driver problem. What nvidia drivers are you guys using?
[mrinal] I'm using NVIDIA driver 260.19.06 on a GeForce 7300 GT. After reading the other thread, it's starting to look more and more like a driver issue. Although I don't see how killing a TCP connection affects the graphics driver...
[rusu] 260.24 here -- I also had 256.35 and 256.53, and I remember having the same problems. I can try on another machine with a different card/driver.
[jfaust] latest apt-gettable drivers (256.35) with latest cturtle debs work fine on my machine. Radu, if you can reproduce on the released drivers, I can take a look on your machine
[rusu] Kai has the same issue on bie (stock NVidia driver install from Ubuntu 10.04). Nathan set up the machine so it's standard.
[rusu] {{{ rusu@bie:~$ dmesg | grep NV [ 0.000000] BIOS-e820: 00000000bf79e000 - 00000000bf7d0000 (ACPI NVS) [ 0.000000] modified: 00000000bf79e000 - 00000000bf7d0000 (ACPI NVS) [ 10.153423] nvidia: module license 'NVIDIA' taints kernel. [ 10.700088] NVRM: loading NVIDIA UNIX x86_64 Kernel Module 256.35 Wed Jun 16 18:42:44 PDT 2010 }}}
[rusu] Yey! I finally made it crash:
{{{
rviz: /opt/ros/cturtle/stacks/visualization_common/ogre/build/ogre_src_v1-7-1/OgreMain/src/OgreFileSystem.cpp:219: virtual Ogre::DataStreamPtr Ogre::FileSystemArchive::open(const Ogre::String&, bool) const: Assertion `ret == 0 && "Problem getting file size"' failed.
Program received signal SIGABRT, Aborted.
0x00007fffef235a75 in ___GI_raise (sig=
function=0x7ffff756f8c0 "virtual Ogre::DataStreamPtr Ogre::FileSystemArchive::open(const Ogre::String&, bool) const") at assert.c:81
at /opt/ros/cturtle/stacks/visualization_common/ogre/build/ogre_src_v1-7-1/OgreMain/src/OgreFileSystem.cpp:219
resourceBeingLoaded=<value optimized out>) at /opt/ros/cturtle/stacks/visualization_common/ogre/build/ogre_src_v1-7-1/OgreMain/src/OgreResourceGroupManager.cpp:696
resourceBeingLoaded=<value optimized out>) at /opt/ros/cturtle/stacks/visualization_common/ogre/build/ogre_src_v1-7-1/OgreMain/src/OgreResourceGroupManager.cpp:746
indexBufferUsage=<value optimized out>, vertexBufferShadowed=<value optimized out>, indexBufferShadowed=<value optimized out>)
at /opt/ros/cturtle/stacks/visualization_common/ogre/build/ogre_src_v1-7-1/OgreMain/src/OgreMeshManager.cpp:123
at /opt/ros/cturtle/stacks/visualization_common/ogre/build/ogre_src_v1-7-1/OgreMain/src/OgreEntity.cpp:2181
at /opt/ros/cturtle/stacks/visualization_common/ogre/build/ogre_src_v1-7-1/OgreMain/src/OgreMovableObject.cpp:471
at /opt/ros/cturtle/stacks/visualization_common/ogre/build/ogre_src_v1-7-1/OgreMain/src/OgreSceneManager.cpp:6487
at /opt/ros/cturtle/stacks/visualization_common/ogre/build/ogre_src_v1-7-1/OgreMain/src/OgreSceneManager.cpp:536
at /opt/ros/cturtle/stacks/visualization_common/ogre_tools/src/ogre_tools/shape.cpp:70
origin=<value optimized out>, parent_node=<value optimized out>, entity=@0x7fffd119b158, scene_node=@0x7fffd119b168, offset_node=@0x7fffd119b170)
at /opt/ros/cturtle/stacks/visualization/rviz/src/rviz/robot/robot_link.cpp:311
at /opt/ros/cturtle/stacks/visualization/rviz/src/rviz/robot/robot_link.cpp:161
at /opt/ros/cturtle/stacks/visualization/rviz/src/rviz/robot/robot.cpp:215
from /opt/ros/cturtle/stacks/visualization/rviz/lib/librviz.so
---Type
}}}
[jfaust] That's what happens when you ctrl-c the publisher? That looks like it was triggered from enabling the robot model display.
[rusu] Yeah, that was on Ctrl+C.
I got another one just by leaving RViz on for a few minutes (during lunch today). When I came back, I got this: {{{
* glibc detected * /opt/ros/cturtle/stacks/visualization/rviz/bin/rviz: malloc(): smallbin double linked list corrupted: 0x00007fffd7a63b90 ***
======= Backtrace: =========
/lib/libc.so.6(+0x775b6)[0x7fffef2795b6]
/lib/libc.so.6(+0x7b8e9)[0x7fffef27d8e9]
/lib/libc.so.6(libc_malloc+0x6e)[0x7fffef27e58e]
/usr/lib64/libwx_baseu-2.8.so.0(_ZN12wxStringBase11AllocBufferEm+0x54)[0x7ffff0448454]
/usr/lib64/libwx_baseu-2.8.so.0(_ZN12wxStringBase16AllocBeforeWriteEm+0x2e)[0x7ffff044981e]
/usr/lib64/libwx_baseu-2.8.so.0(_ZN8wxString11GetWriteBufEm+0x9)[0x7ffff0449899]
/usr/lib64/libwx_baseu-2.8.so.0(_ZN8wxString9FromAsciiEPKc+0xbb)[0x7ffff044af2b]
/opt/ros/cturtle/stacks/visualization/rviz/lib/librviz.so(_ZN4rviz14StatusProperty9setStatusENS_13statuslevels11StatusLevelERKSsS4+0x236)[0x7ffff21a97b6]
/opt/ros/cturtle/stacks/visualization/rviz/lib/librviz.so(_ZN4rviz7Display9setStatusENS_13statuslevels11StatusLevelERKSsS4+0x5d)[0x7ffff221996d]
/opt/ros/cturtle/stacks/visualization/rviz/lib/libdefault_plugin.so(_ZN4rviz14PointCloudBase12updateStatusEv+0x20b)[0x7fffc84c5edb]
/opt/ros/cturtle/stacks/visualization/rviz/lib/libdefault_plugin.so(_ZN4rviz14PointCloudBase6updateEff+0x4e6)[0x7fffc84c6466]
/opt/ros/cturtle/stacks/visualization/rviz/lib/librviz.so(_ZN4rviz20VisualizationManager8onUpdateER12wxTimerEvent+0x204)[0x7ffff21f73d4]
/usr/lib64/libwx_baseu-2.8.so.0(_ZN12wxEvtHandler21ProcessEventIfMatchesERK21wxEventTableEntryBasePS_R7wxEvent+0x50)[0x7ffff0482ff0]
/usr/lib64/libwx_baseu-2.8.so.0(_ZN12wxEvtHandler23SearchDynamicEventTableER7wxEvent+0x4f)[0x7ffff048315f]
/usr/lib64/libwx_baseu-2.8.so.0(_ZN12wxEvtHandler12ProcessEventER7wxEvent+0x92)[0x7ffff0484072]
/usr/lib64/libwx_gtk2u_core-2.8.so.0(_ZN11wxTimerBase6NotifyEv+0x68)[0x7ffff0e0b3f8]
/usr/lib64/libwx_gtk2u_core-2.8.so.0(+0x1ebe8b)[0x7ffff0d16e8b]
/lib/libglib-2.0.so.0(+0x3f09b)[0x7fffec98c09b]
/lib/libglib-2.0.so.0(g_main_context_dispatch+0x1f2)[0x7fffec98b8c2]
/lib/libglib-2.0.so.0(+0x42748)[0x7fffec98f748]
/lib/libglib-2.0.so.0(g_main_loop_run+0x195)[0x7fffec98fc55]
/usr/lib64/libgtk-x11-2.0.so.0(gtk_main+0xa7)[0x7fffeed1bbb7]
/usr/lib64/libwx_gtk2u_core-2.8.so.0(_ZN11wxEventLoop3RunEv+0x48)[0x7ffff0d0e558]
/usr/lib64/libwx_gtk2u_core-2.8.so.0(_ZN9wxAppBase8MainLoopEv+0x4b)[0x7ffff0d932eb]
/usr/lib64/libwx_baseu-2.8.so.0(_Z7wxEntryRiPPw+0x4c)[0x7ffff042947c]
/opt/ros/cturtle/stacks/visualization/rviz/bin/rviz(main+0x12)[0x413b62]
/lib/libc.so.6(__libc_start_main+0xfd)[0x7fffef220c4d]
/opt/ros/cturtle/stacks/visualization/rviz/bin/rviz[0x413a39]
======= Memory map: ========
00400000-00422000 r-xp 00000000 08:01 2662865 /opt/ros/cturtle/stacks/visualization/rviz/bin/rviz
00622000-00623000 r--p 00022000 08:01 2662865 /opt/ros/cturtle/stacks/visualization/rviz/bin/rviz
00623000-00624000 rw-p 00023000 08:01 2662865 /opt/ros/cturtle/stacks/visualization/rviz/bin/rviz
00624000-17194000 rw-p 00000000 00:00 0 [heap]
40000000-40066000 rw-p 00000000 00:05 1151 /dev/zero
40066000-40069000 r-xs 00000000 08:01 4219849 /tmp/gltkeFnd (deleted)
7fffa8000000-7fffa9069000 rw-p 00000000 00:00 0
7fffa9069000-7fffac000000 ---p 00000000 00:00 0
7fffac000000-7fffafd96000 rw-p 00000000 00:00 0
7fffafd96000-7fffb0000000 ---p 00000000 00:00 0
7fffb0000000-7fffb326e000 rw-p 00000000 00:00 0
7fffb326e000-7fffb4000000 ---p 00000000 00:00 0
7fffb7a60000-7fffb8000000 rw-s 0bee4000 00:05 5406 /dev/nvidia0
7fffb8000000-7fffbbfe4000 rw-p 00000000 00:00 0
7fffbbfe4000-7fffbc000000 ---p 00000000 00:00 0
7fffbd0cf000-7fffbd1cf000 rw-p 00000000 00:00 0
7fffbd1cf000-7fffbd76f000 rw-s 1355a000 00:05 5406 /dev/nvidia0
7fffbd76f000-7fffbd96f000 rw-s 0428b000 00:05 5406 /dev/nvidia0
7fffbd96f000-7fffbdf0f000 rw-s 8a94e000 00:05 5406 /dev/nvidia0
7fffbdf0f000-7fffbe4af000 rw-s aa728000 00:05 5406 /dev/nvidia0
7fffbe4af000-7fffbea4f000 rw-s c6d04000 00:05 5406 /dev/nvidia0
7fffbea4f000-7fffbec4f000 rw-s 3489e000 00:05 5406 /dev/nvidia0
7fffbec4f000-7fffbf627000 rw-s 8aab0000 00:05 5406 /dev/nvidia0
7fffbf627000-7fffc0000000 rw-p 00000000 00:00 0
7fffc0000000-7fffc3fb0000 rw-p 00000000 00:00 0
7fffc3fb0000-7fffc4000000 ---p 00000000 00:00 0
7fffc43f2000-7fffc45f2000 rw-s bbafd000 00:05 5406 /dev/nvidia0
7fffc45f2000-7fffc4fcb000 rw-p 00000000 00:00 0
7fffc5406000-7fffc5dde000 rw-s 7c9b1000 00:05 5406 /dev/nvidia0
7fffc6073000-7fffc6613000 rw-s 0808d000 00:05 5406 /dev/nvidia0
7fffc6613000-7fffc6614000 ---p 00000000 00:00 0
7fffc6614000-7fffc6e14000 rwxp 00000000 00:00 0
7fffc6e14000-7fffc6e21000 r-xp 00000000 08:01 559998 /usr/lib/libgomp.so.1.0.0
7fffc6e21000-7fffc7020000 ---p 0000d000 08:01 559998 /usr/lib/libgomp.so.1.0.0
7fffc7020000-7fffc7021000 r--p 0000c000 08:01 559998 /usr/lib/libgomp.so.1.0.0
7fffc7021000-7fffc7022000 rw-p 0000d000 08:01 559998 /usr/lib/libgomp.so.1.0.0
7fffc7022000-7fffc7039000 r-xp 00000000 08:01 2547758 /opt/ros/cturtle/stacks/common/actionlib/lib/libactionlib.so
7fffc7039000-7fffc7238000 ---p 00017000 08:01 2547758 /opt/ros/cturtle/stacks/common/actionlib/lib/libactionlib.so
7fffc7238000-7fffc7239000 r--p 00016000 08:01 2547758 /opt/ros/cturtle/stacks/common/actionlib/lib/libactionlib.so
7fffc7239000-7fffc723a000 rw-p 00017000 08:01 2547758 /opt/ros/cturtle/stacks/common/actionlib/lib/libactionlib.so
7fffc723a000-7fffc7356000 r-xp 00000000 08:01 2608430 /opt/ros/cturtle/stacks/physics_ode/opende/opende/lib/libode.so.1.1.1
7fffc7356000-7fffc7556000 ---p 0011c000 08:01 2608430 /opt/ros/cturtle/stacks/physics_ode/opende/opende/lib/libode.so.1.1.1
7fffc7556000-7fffc7559000 r--p 0011c000 08:01 2608430 /opt/ros/cturtle/stacks/physics_ode/opende/opende/lib/libode.so.1.1.1
7fffc7559000-7fffc755b000 rw-p 0011f000 08:01 2608430 /opt/ros/cturtle/stacks/physicsode/opende/opende/lib/libode.so.1.1.1
Program received signal SIGABRT, Aborted.
0x00007fffef235a75 in GI_raise (sig=
at /opt/ros/cturtle/stacks/visualization/rviz/src/rviz/properties/property.cpp:251
at /opt/ros/cturtle/stacks/visualization/rviz/src/rviz/default_plugin/point_cloud_base.cpp:770
(gdb) r }}}
[jfaust] What was it doing when this happened? Were you connected to a robot? What was your config?
[rusu] I attached the rviz vcg config. We were connected to pri, and it was just displaying point cloud data in the background (screen locked while I went to lunch).
I'll come grab you next time it happens, maybe we catch a live one.
[watts] This looks related to #4463, which is probably a graphics driver problem. I was able to duplicate that on my machine pretty easily. If you want to try my desktop, feel free.
[jfaust] This is definitely a driver problem. Installing the latest beta drivers (260.19.12) causes any subscriber that's has libGL.so loaded to crash when a publisher disconnects.
{{{ roscd roscpp_tutorials LD_PRELOAD=libGL.so bin/listener }}}
{{{ roscd roscpp_tutorials bin/talker
[kartikmohta] Just to note, this problem is fixed in driver version 260.19.21.
I'm using the ros-cturtle debs on Lucid Lynx. I have rviz displaying a robot model, and tf transforms are being published by robot_state_publisher. When I kill robot_state_publisher, rviz crashes with no error messages other than "Segmentation fault". The problem is not reproducible under gdb. Neither is it reproducible reliably under valgrind, but I got some interesting valgrind complaints just once when trying to reproduce it. (attached)
Let me know if you have further ideas on reproducing this problem, or any further information that might help.
trac data: