[Crashes!] Gazebo 7 simulation crashes!

hejia-zhang commented 6 years ago

Description

The gazebo crashes in my RL training process for sawyer.
After I found the minimal code to reproduce the crash, I found that the crash happens when I repeatedly call set_joint_positions and rospy.wait_for_message(...). And the crash will happen after about 1500 iterations.
The minimal code to reproduce the crash is as follows:

import rospy
import intera_interface
from gazebo_msgs.msg import (
    ModelStates
)

class BugReproducer(object):
    def __init__(self, limb="right", hover_distance = 0.15, tip_name="right_gripper_tip"):
        self._limb = intera_interface.Limb(limb)
        self._rs = intera_interface.RobotEnable(intera_interface.CHECK_VERSION)
        self._rs.enable()

    def clean_shutdown(self):
        print("Exiting...")
        if self._rs.state().enabled:
            print("Disabling robot...")
            self._rs.disable()

rospy.init_node("bug_reproducer")
br = BugReproducer()
rospy.on_shutdown(br.clean_shutdown)
joint_angles = {'right_j0': -0.041662954890248294,
                'right_j1': -1.0258291091425074,
                'right_j2': 0.0293680414401436,
                'right_j3': 2.17518162913313,
                'right_j4':  -0.06703022873354225,
                'right_j5': 0.3968371433926965,
                'right_j6': 1.7659649178699421}
while True:
    br._limb.set_joint_positions(joint_angles)
    rospy.wait_for_message("/gazebo/model_states", ModelStates, timeout=5)

And the backtrace of the crash is:

#0  gazebo::physics::World::Update (this=this@entry=0x10cd580)
    at /build/gazebo-nhSAPd/gazebo-7.0.0+dfsg/gazebo/physics/World.cc:740
#1  0x00007ffff62d89af in gazebo::physics::World::Step (this=this@entry=0x10cd580)
    at /build/gazebo-nhSAPd/gazebo-7.0.0+dfsg/gazebo/physics/World.cc:672
#2  0x00007ffff62d8e25 in gazebo::physics::World::RunLoop (this=0x10cd580)
    at /build/gazebo-nhSAPd/gazebo-7.0.0+dfsg/gazebo/physics/World.cc:481
#3  0x00007ffff40d85d5 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.58.0
#4  0x00007ffff65ed6ba in start_thread (arg=0x7fff377fe700) at pthread_create.c:333
#5  0x00007ffff6bef41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Environment Parameters

Operating System Version: Ubuntu 16.04
ROS Version: Kinetic
Gazebo Version: Gazebo 7.0.0

IanTheEngineer commented 6 years ago

Hmm, I can replicate your issue, however, I think we can simplify it even further by removing Sawyer from the equation:

#!/usr/bin/env python

###  tester.py
import rospy
from gazebo_msgs.msg import ModelStates

rospy.init_node("bug_reproducer")
while not rospy.is_shutdown():
    rospy.wait_for_message("/gazebo/model_states", ModelStates, timeout=5)

then two terminals:

1) $ roslaunch gazebo_ros empty_world.launch debug:=true

2) $ python tester.py

produce the same result:

Thread 37 "gzserver" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fff3abfc700 (LWP 18932)]
0x00007ffff60a2328 in ?? () from /usr/lib/x86_64-linux-gnu/libgazebo_physics.so.7
(gdb) bt full
#0  0x00007ffff60a2328 in ?? () from /usr/lib/x86_64-linux-gnu/libgazebo_physics.so.7
No symbol table info available.
#1  0x00007ffff608b959 in gazebo::physics::World::Update() () from /usr/lib/x86_64-linux-gnu/libgazebo_physics.so.7
No symbol table info available.
#2  0x00007ffff609b0df in gazebo::physics::World::Step() () from /usr/lib/x86_64-linux-gnu/libgazebo_physics.so.7
No symbol table info available.
#3  0x00007ffff609b555 in gazebo::physics::World::RunLoop() () from /usr/lib/x86_64-linux-gnu/libgazebo_physics.so.7
No symbol table info available.
#4  0x00007ffff3e335d5 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.58.0
No symbol table info available.
#5  0x00007ffff63c06ba in start_thread (arg=0x7fff3abfc700) at pthread_create.c:333
__res = <optimized out>
pd = 0x7fff3abfc700
now = <optimized out>
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140734179034880, -6835587323277935288, 0, 140737488339455, 140734179035584, 16226272, 6835735209336044872,
6835604363778522440}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call = <optimized out>
pagesize_m1 = <optimized out>
sp = <optimized out>
freesize = <optimized out>
__PRETTY_FUNCTION__ = "start_thread"
#6  0x00007ffff69ca41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

As this doesn't seem to be specific to the sawyer_simulator, I would recommend to report this on Gazebo's bitbucket issue tracker: https://bitbucket.org/osrf/gazebo/issues?status=new&status=open

I am not super familiar with the rospy.wait_for_message function that seems to be causing this issue, as I'm use to the subscriber-callback pattern for ROS nodes. As a workaround, I would recommend either adding a sleep in your loop to prevent pinging Gazebo too quickly, or just create a callback to wait for messages the traditional way: http://wiki.ros.org/rospy/Overview/Publishers%20and%20Subscribers#Subscribing_to_a_topic

IanTheEngineer commented 6 years ago

@hjzh4 I'm going to close this ticket with the expectation that this will be brought upstream to the folks working on Gazebo. Please let me know if there are additional Sawyer-specific details and we can reopen the ticket.

hejia-zhang commented 6 years ago

@IanTheEngineer Okay, thanks for your help!

RethinkRobotics / sawyer_simulator

[Crashes!] Gazebo 7 simulation crashes! #38

Description

Environment Parameters