Upper body detector crashes quite regularly in simulation

[kunzel]

The people detection works awesome in simulation! But unfortunately, the upper body detector crashes often without any notice. The mentioned log file also does not exist. Any idea, what could cause the problem? It often happens when the human is not in sight any more.

core service [/rosout] found
process[odom2visual-1]: started with pid [3401]
process[ground_plane-2]: started with pid [3417]
process[upper_body_detector-3]: started with pid [3440]
process[pedestrian_tracking-4]: started with pid [3515]
process[pedestrian_localisation-5]: started with pid [3631]
process[save_people_locations-6]: started with pid [3694]
[INFO] [WallTime: 1406106065.266644] Waiting for services...
[INFO] [WallTime: 1406106065.280372] Done
[upper_body_detector-3] process has died [pid 3440, exit code -11, cmd /home/lars/catkin_ws/devel/lib/strands_upper_body_detector/upper_body_detector __name:=upper_body_detector __log:=/home/lars/.ros/log/c12c7bb8-1240-11e4-b8af-f01faf28c879/upper_body_detector-3.log].
log file: /home/lars/.ros/log/c12c7bb8-1240-11e4-b8af-f01faf28c879/upper_body_detector-3*.log

[cdondrup]

Above pull request switches the output of the nodes to log files instead of screen. Since I wasn't able to run the same simulator setup on my machine, didn't know what to run, please check this out and try to reproduce the error. Then post the logfile output here please.

Also, what did you run to get the ppl perception to work in simulation? Looks amazing!

[kunzel]

@cdondrup, I've just tested it with enhanced logging. Now a log file is created but the contents is still empty? Do I have to compile it in a different way?

To get everything running in simulation you have to launch also: roslaunch strands_morse generate_camera_topics.launch after starting the simulator.

In your MORSE build script you have to use the following option:

robot = Scitosa5(with_cameras = Scitosa5.WITH_OPENNI) [..] human = Human() human.use_world_camera()

This will generate all topics of the camera that are also provided by the real robot. (see here: https://github.com/strands-project/strands_morse/pull/68)

[cdondrup]

Ok, I tried to debug it a little but this is hell. As soon as you enable debug symbols it crashes due to various assertions failures. Here is the backtrace I got without debug symbols... so rather useless:

Program received signal SIGSEGV, Segmentation fault.
_int_free (av=0x7ffff5023720, p=0x809350, have_lock=0) at malloc.c:4117
4117    malloc.c: No such file or directory.
(gdb) bt
#0  _int_free (av=0x7ffff5023720, p=0x809350, have_lock=0) at malloc.c:4117
#1  0x0000000000473168 in Detector::ProcessFrame(Camera const&, Matrix<double> const&, PointCloud const&, Matrix<double> const&, Vector<Vector<double> >&) ()
#2  0x000000000043220b in callback(boost::shared_ptr<sensor_msgs::Image_<std::allocator<void> > const> const&, boost::shared_ptr<sensor_msgs::Image_<std::allocator<void> > const> const&, boost::shared_ptr<strands_perception_people_msgs::GroundPlane_<std::allocator<void> > const> const&, boost::shared_ptr<sensor_msgs::CameraInfo_<std::allocator<void> > const> const&) ()
#3  0x000000000043ca81 in boost::detail::function::void_function_obj_invoker9<boost::function<void (boost::shared_ptr<sensor_msgs::Image_<std::allocator<void> > const> const&, boost::shared_ptr<sensor_msgs::Image_<std::allocator<void> > const> const&, boost::shared_ptr<strands_perception_people_msgs::GroundPlane_<std::allocator<void> > const> const&, boost::shared_ptr<sensor_msgs::CameraInfo_<std::allocator<void> > const> const&, boost::shared_ptr<message_filters::NullType const> const&, boost::shared_ptr<message_filters::NullType const> const&, boost::shared_ptr<message_filters::NullType const> const&, boost::shared_ptr<message_filters::NullType const> const&, boost::shared_ptr<message_filters::NullType const> const&)>, void, boost::shared_ptr<sensor_msgs::Image_<std::allocator<void> > const>, boost::shared_ptr<sensor_msgs::Image_<std::allocator<void> > const>, boost::shared_ptr<strands_perception_people_msgs::GroundPlane_<std::allocator<void> > const>, boost::shared_ptr<sensor_msgs::CameraInfo_<std::allocator<void> > const>, boost::shared_ptr<message_filters::NullType const>, boost::shared_ptr<message_filters::NullType const>, boost::shared_ptr<message_filters::NullType const>, boost::shared_ptr<message_filters::NullType const>, boost::shared_ptr<message_filters::NullType const> >::invoke(boost::detail::function::function_buffer&, boost::shared_ptr<sensor_msgs::Image_<std::allocator<void> > const>, boost::shared_ptr<sensor_msgs::Image_<std::allocator<void> > const>, boost::shared_ptr<strands_perception_people_msgs::GroundPlane_<std::allocator<void> > const>, boost::shared_ptr<sensor_msgs::CameraInfo_<std::allocator<void> > const>, boost::shared_ptr<message_filters::NullType const>, boost::shared_ptr<message_filters::NullType const>, boost::shared_ptr<message_filters::NullType const>, boost::shared_ptr<message_filters::NullType const>, boost::shared_ptr<message_filters::NullType const>) ()
#4  0x000000000045f742 in boost::function9<void, boost::shared_ptr<sensor_msgs::Image_<std::allocator<void> > const>, boost::shared_ptr<sensor_msgs::Image_<std::allocator<void> > const>, boost::shared_ptr<strands_perception_people_msgs::GroundPlane_<std::allocator<void> > const>, boost::shared_ptr<sensor_msgs::CameraInfo_<std::allocator<void> > const>, boost::shared_ptr<message_filters::NullType const>, boost::shared_ptr<message_filters::NullType const>, boost::shared_ptr<message_filters::NullType const>, boost::shared_ptr<message_filters::NullType const>, boost::shared_ptr<message_filters::NullType const> >::operator()(boost::shared_ptr<sensor_msgs::Image_<std::allocator<void> > const>, boost::shared_ptr<sensor_msgs::Image_<std::allocator<void> > const>, boost::shared_ptr<strands_perception_people_msgs::GroundPlane_<std::allocator<void> > const>, boost::shared_ptr<sensor_msgs::CameraInfo_<std::allocator<void> > const>, boost::shared_ptr<message_filters::NullType const>, boost::shared_ptr<message_filters::NullType const>, boost::shared_ptr<message_filters::NullType const>, boost::shared_ptr<message_filters::NullType const>, boost::shared_ptr<message_filters::NullType const>) const ()
#5  0x00000000004602d3 in message_filters::CallbackHelper9T<boost::shared_ptr<sensor_msgs::Image_<std::allocator<void> > const> const&, boost::shared_ptr<sensor_msgs::Image_<std::allocator<void> > const> const&, boost::shared_ptr<strands_perception_people_msgs::GroundPlane_<std::allocator<void> > const> const&, boost::shared_ptr<sensor_msgs::CameraInfo_<std::allocator<void> > const> const&, boost::shared_ptr<message_filters::NullType const> const&, boost::shared_ptr<message_filters::NullType const> const&, boost::shared_ptr<message_filters::NullType const> const&, boost::shared_ptr<message_filters::NullType const> const&, boost::shared_ptr<message_filters::NullType const> const&>::call(bool, ros::MessageEvent<sensor_msgs::Image_<std::allocator<void> > const> const&, ros::MessageEvent<sensor_msgs::Image_<std::allocator<void> > const> const&, ros::MessageEvent<strands_perception_people_msgs::GroundPlane_<std::allocator<void> > const> const&, ros::MessageEvent<sensor_msgs::CameraInfo_<std::allocator<void> > const> const&, ros::MessageEvent<message_filters::NullType const> const&, ros::MessageEvent<message_filters::NullType const> const&, ros::MessageEvent<message_filters::NullType const> const&, ros::MessageEvent<message_filters::NullType const> const&, ros::MessageEvent<message_filters::NullType const> const&) ()
#6  0x00000000004583ba in message_filters::sync_policies::ApproximateTime<sensor_msgs::Image_<std::allocator<void> >, sensor_msgs::Image_<std::allocator<void> >, strands_perception_people_msgs::GroundPlane_<std::allocator<void> >, sensor_msgs::CameraInfo_<std::allocator<void> >, message_filters::NullType, message_filters::NullType, message_filters::NullType, message_filters::NullType, message_filters::NullType>::publishCandidate() ()
#7  0x000000000045ba98 in message_filters::sync_policies::ApproximateTime<sensor_msgs::Image_<std::allocator<void> >, sensor_msgs::Image_<std::allocator<void> >, strands_perception_people_msgs::GroundPlane_<std::allocator<void> >, sensor_msgs::CameraInfo_<std::allocator<void> >, message_filters::NullType, message_filters::NullType, message_filters::NullType, message_filters::NullType, message_filters::NullType>::process() ()
#8  0x000000000045e225 in void message_filters::sync_policies::ApproximateTime<sensor_msgs::Image_<std::allocator<void> >, sensor_msgs::Image_<std::allocator<void> >, strands_perception_people_msgs::GroundPlane_<std::allocator<void> >, sensor_msgs::CameraInfo_<std::allocator<void> >, message_filters::NullType, message_filters::NullType, message_filters::NullType, message_filters::NullType, message_filters::NullType>::add<1>(boost::mpl::at_c<boost::mpl::vector<ros::MessageEvent<sensor_msgs::Image_<std::allocator<void> > const>, ros::MessageEvent<sensor_msgs::Image_<std::allocator<void> > const>, ros::MessageEvent<strands_perception_people_msgs::GroundPlane_<std::allocator<void> > const>, ros::MessageEvent<sensor_msgs::CameraInfo_<std::allocator<void> > const>, ros::MessageEvent<message_filters::NullType const>, ros::MessageEvent<message_filters::NullType const>, ros::MessageEvent<message_filters::NullType const>, ros::MessageEvent<message_filters::NullType const>, ros::MessageEvent<message_filters::NullType const>, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, 1>::type const&) ()
#9  0x0000000000443638 in message_filters::CallbackHelper1T<ros::MessageEvent<sensor_msgs::Image_<std::allocator<void> > const> const&, sensor_msgs::Image_<std::allocator<void> > >::call(ros::MessageEvent<sensor_msgs::Image_<std::allocator<void> > const> const&, bool) ()
#10 0x00000000004402df in image_transport::SubscriberFilter::cb(boost::shared_ptr<sensor_msgs::Image_<std::allocator<void> > const> const&) ()
#11 0x00007fffe8ac7227 in image_transport::RawSubscriber::internalCallback(boost::shared_ptr<sensor_msgs::Image_<std::allocator<void> > const> const&, boost::function<void (boost::shared_ptr<sensor_msgs::Image_<std::allocator<void> > const> const&)> const&) () from /opt/ros/hydro/lib//libimage_transport_plugins.so
#12 0x00007ffff756280a in boost::detail::function::void_function_obj_invoker1<boost::function<void (boost::shared_ptr<sensor_msgs::Image_<std::allocator<void> > const> const&)>, void, boost::shared_ptr<sensor_msgs::Image_<std::allocator<void> > const> >::invoke(boost::detail::function::function_buffer&, boost::shared_ptr<sensor_msgs::Image_<std::allocator<void> > const>) () from /opt/ros/hydro/lib/libimage_transport.so
#13 0x00007fffe8ace525 in ros::SubscriptionCallbackHelperT<boost::shared_ptr<sensor_msgs::Image_<std::allocator<void> > const> const&, void>::call(ros::SubscriptionCallbackHelperCallParams&) ()
   from /opt/ros/hydro/lib//libimage_transport_plugins.so
#14 0x00007ffff70c2e57 in ros::SubscriptionQueue::call() () from /opt/ros/hydro/lib/libroscpp.so
#15 0x00007ffff70773e9 in ros::CallbackQueue::callOneCB(ros::CallbackQueue::TLS*) () from /opt/ros/hydro/lib/libroscpp.so
#16 0x00007ffff7078eab in ros::CallbackQueue::callAvailable(ros::WallDuration) () from /opt/ros/hydro/lib/libroscpp.so
#17 0x00007ffff70c67c8 in ros::SingleThreadedSpinner::spin(ros::CallbackQueue*) () from /opt/ros/hydro/lib/libroscpp.so
#18 0x00007ffff70abe1b in ros::spin() () from /opt/ros/hydro/lib/libroscpp.so
#19 0x000000000042cf36 in main ()

I narrowed it down to the Detector::ComputeFreespace function.

The thing is, if it works on the real robot but not in simulation there must be a difference between the produced data. @nilsbore can you think of anything?

[nilsbore]

I'm on vacation this week so I might not be able to help out much more now. It might be that the images are coming at 20 Hz in simulation vs commonly 30 Hz on the robots.

[kunzel]

Thanks, @nilsbore. I tried running the camera at 30 Hz, but it did not help. The upper_body_detector is still crashing.

[nilsbore]

Ok, hmm, the 20Hz is set at two places, in the ScitosA5 python file for the RGB camera and in src/generate_topics/generate_disparity.cpp for the depth. I'll turn the second one into a parameter when I have time.

[nilsbore]

Otherwise, there might be something with how the respective image_raw are encoded. At least for the RGB one, I think the one coming from Morse is rgb instead of bgr for the openni stack.

[cdondrup]

Yes it is. That's why the bounding box is blue in simulation and red on the real robot but the upper body detector only uses the color image for the visualisation. That shouldn't be the problem. But fixing this would be good in general.

[ferdianjovan]

I ran the simulation several times and the upper body detection never crashed. The condition is that the robot stands still observing people around. The longest running simulation was more than 30 mins and it worked fine. However, when I ran the simulation with the robot following a person, it crashed several times.

[filipetrocadoferreira]

I have similar error using upper body detection with a custom depth image produced from a stereo camera setup.