search

MIT-SPARK / Hydra

BSD 2-Clause "Simplified" License
525 stars 65 forks source link

[QUESTION] Run uHumans dataset but no scene graph is built #17

Closed glennliu closed 9 months ago

glennliu commented 1 year ago

Hi Nathan

I compiled the project successfully and run it with uHuman2_apartment_s1_00h.bag. I strictly followed the instruction and have not changed any parameters.

Here are the output after roslaunch hydra_dsg_builder uhumans2_incremental_dsg.launch start_visualizer:=true,

process[pose_graph_publisher_node-14]: started with pid [5732]
process[incremental_dsg_builder_node/pgmo/posegraph_viewer-15]: started with pid [5737]
[ WARN] [1664427109.764515173]: PoseGraphPublisher waiting for subscriber...
[ INFO] [1664427109.765233956]: Initializing pose graph visualizer
[ INFO] [1664427109.805238745]: rviz version 1.13.18
[ INFO] [1664427109.805274556]: compiled against Qt version 5.9.5
[ INFO] [1664427109.805285194]: compiled against OGRE version 1.9.0 (Ghadamon)
[ INFO] [1664427109.817109855]: Forcing OpenGl version 0.
XmbTextListToTextProperty result code -2
XmbTextListToTextProperty result code -2
[ INFO] [1664427109.912558213]: Logging backend graph to /home/lch/mit_ws/catkin_ws/src/Hydra/hydra_dsg_builder/output/uhumans2/office/backend
[ INFO] [1664427109.918640681]: Logging output to: /home/lch/mit_ws/catkin_ws/src/Hydra/hydra_dsg_builder/output/uhumans2/office/pgmo
[ WARN] [1664427109.920612979]: MeshFrontend: Skipping callback registration
[ INFO] [1664427109.920758069]: Initialized MeshFrontend.
[ INFO] [1664427109.920776062]: Logging frontend graph to /home/lch/mit_ws/catkin_ws/src/Hydra/hydra_dsg_builder/output/uhumans2/office/frontend
[ WARN] [1664427110.169361286]: PoseGraphPublisher waiting for transform: left_cam -> world
[ INFO] [1664427110.297691937]: Stereo is NOT SUPPORTED
[ INFO] [1664427110.297737212]: OpenGL device: GeForce GTX 1080 Ti/PCIe/SSE2
[ INFO] [1664427110.297752060]: OpenGl version: 4.6 (GLSL 4.6).
XmbTextListToTextProperty result code -2
XmbTextListToTextProperty result code -2
[ WARN] [1664427114.466186140, 3.530520571]: using timer!
XmbTextListToTextProperty result code -2
[ INFO] [1664427116.238928121, 4.416039183]: New prefix s detected when adding new mesh edges and nodes. 
[ INFO] [1664427116.339645156, 4.466376633]: DeformationGraph: Re-calculating all mesh vertices in deformMesh. 
[ERROR] [1664427116.339818027, 4.466376633]: Not enough valid control points in octree to interpolate point. 
[ERROR] [1664427116.339883765, 4.466376633]: Not enough valid control points in octree to interpolate point. 
*** Aborted at 1664427116 (unix time) try "date -d @1664427116" if you are using GNU date ***
PC: @                0x0 (unknown)
*** SIGSEGV (@0x2d) received by PID 5730 (TID 0x7f8c5dffb700) from PID 45; stack trace: ***
    @     0x7f8c8e064040 (unknown)
    @     0x7f8c8e0bca6d cfree
    @     0x7f8c8d47818a pcl::octree::OctreeBase<>::deleteBranchChild()
    @     0x7f8c8d4781b8 pcl::octree::OctreeBase<>::deleteBranchChild()
    @     0x7f8c8d4781f0 pcl::octree::OctreeBase<>::~OctreeBase()
    @     0x7f8c9036a860 boost::detail::sp_counted_impl_p<>::dispose()
    @     0x55785e3ccf7a boost::detail::sp_counted_base::release()
    @     0x7f8c903a5c18 kimera_pgmo::deformPointsWithTimeCheck<>()
    @     0x7f8c9039e0ef kimera_pgmo::DeformationGraph::deformMesh()
    @     0x7f8c9039e7f9 kimera_pgmo::DeformationGraph::deformMesh()
    @     0x7f8c90c514d0 hydra::incremental::DsgBackend::updateDsgMesh()
    @     0x7f8c90c579ea hydra::incremental::DsgBackend::runPgmo()
    @     0x7f8c8e6eb6df (unknown)
    @     0x7f8c8bc946db start_thread
    @     0x7f8c8e14671f clone
================================================================================REQUIRED process [incremental_dsg_builder_node-13] has died!
process has died [pid 5730, exit code -11, cmd /home/lch/mit_ws/catkin_ws/devel/lib/hydra_dsg_builder/hydra_dsg_builder_incremental_node --minloglevel=3 -v=3 ~voxblox_mesh:=/hydra_topology_node/active_mesh ~active_places:=/hydra_topology_node/active_layer ~/bow_vectors:=/kimera_vio_ros/bow_query frame_registration:=/kimera_vio_ros/kimera_vio_ros_node/register_lcd_frames __name:=incremental_dsg_builder_node __log:=/home/lch/.ros/log/6d41692a-3fb2-11ed-ad53-1831bfbc9e5b/incremental_dsg_builder_node-13.log].
log file: /home/lch/.ros/log/6d41692a-3fb2-11ed-ad53-1831bfbc9e5b/incremental_dsg_builder_node-13*.log
Initiating shutdown!
================================================================================
[incremental_dsg_builder_node/pgmo/posegraph_viewer-15] killing on exit

The program crashed around 2 seconds after the bag has been run. Not sure is it because the rostime in the bag is significantly different from my current rostime.
To debug, I comment out the launch of hydra_topology/hydra_topology.launch, as I want to confirm the frontend scene graph has been created correctly. Then, the program does not crash anymore and I can see the input sensor data: segmentation images, depth, semantic point cloud. But I cannot see the reconstructed semantic meshes, not to mention object and place markers. An rviz screenshot can be found here,

rviz_screenshot_2022_09_29-13_01_56

It looks like the scene graph cannot be constructed. It should be a front-end problem.

Can you help me with that?

Thanks

hwan0806 commented 1 year ago

First, thank you for wonderful work.

I have a similar issue adapting uHuman2_apartment or subway dataset. Like above question, i exactly followed the instruction of uHumans2_office_s1_00h version and the mapping result was similar except meshed map.

Screenshot from 2022-10-01 00-06-48

I thinked some parameters in uhumans2_incremental_dsg.launch should be changed for other datasets and convert below arguments to appropriate one in kimera_semantics_ros/cfg folder.

https://github.com/MIT-SPARK/Hydra/blob/7ccbadeb14cf73786e4ac88de7ef9a57cdb7459e/hydra_dsg_builder/launch/uhumans2_incremental_dsg.launch#L5-L9 https://github.com/MIT-SPARK/Hydra/blob/7ccbadeb14cf73786e4ac88de7ef9a57cdb7459e/hydra_topology/launch/hydra_topology_uhumans2.launch#L12-L14

After that, hydra package is not launched for below error messages. 

[ INFO] [1664551953.114403717]: Initializing pose graph visualizer
[ WARN] [1664551953.116088151]: PoseGraphPublisher waiting for subscriber...
F1001 00:32:33.176373 328674 color.cpp:51] Check failed: loop->size() == 6 (8 vs. 6) Row 1 is invalid.
*** Check failure stack trace: ***
    @     0x7f793f6ad1c3  google::LogMessage::Fail()
[ INFO] [1664551953.179036916]: Logging backend graph to /home/gil/sparoLab/open_source/Graph_slam/src/hydra/hydra_dsg_builder/output/uhumans2/apartment/backend
    @     0x7f793f6b225b  google::LogMessage::SendToLog()
    @     0x7f793f6acebf  google::LogMessage::Flush()
    @     0x7f793f6ad6ef  google::LogMessageFatal::~LogMessageFatal()
    @     0x7f793e3073ab  kimera::SemanticLabel2Color::SemanticLabel2Color()
    @     0x7f793faee0e6  kimera::getSemanticTsdfIntegratorConfigFromRosParam()
    @     0x7f793faf8cb5  kimera::SemanticTsdfServer::SemanticTsdfServer()
[ INFO] [1664551953.183147875]: Logging output to: /home/gil/sparoLab/open_source/Graph_slam/src/hydra/hydra_dsg_builder/output/uhumans2/apartment/pgmo
    @     0x7f793faf945e  kimera::SemanticTsdfServer::SemanticTsdfServer()
    @     0x56209002d449  hydra::topology::TopologyServer<>::setupLayers()
    @     0x56209002dd64  hydra::topology::TopologyServer<>::TopologyServer()
    @     0x562090017a3b  main
    @     0x7f793f1b7083  __libc_start_main
    @     0x562090017c9e  _start
[ WARN] [1664551953.184279743]: MeshFrontend: Skipping callback registration
[ INFO] [1664551953.185095673]: Initialized MeshFrontend.
[ INFO] [1664551953.185166448]: Logging frontend graph to /home/gil/sparoLab/open_source/Graph_slam/src/hydra/hydra_dsg_builder/output/uhumans2/apartment/frontend
[ INFO] [1664551953.187218466]: rviz version 1.14.19
[ INFO] [1664551953.187237556]: compiled against Qt version 5.12.8
[ INFO] [1664551953.187241696]: compiled against OGRE version 1.9.0 (Ghadamon)
F1001 00:32:33.199155 328688 color.cpp:51] Check failed: loop->size() == 6 (8 vs. 6) Row 1 is invalid.
*** Check failure stack trace: ***
[ INFO] [1664551953.200407565]: Forcing OpenGl version 0.
    @     0x7f41c032e1c3  google::LogMessage::Fail()
    @     0x7f41c033325b  google::LogMessage::SendToLog()
    @     0x7f41c032debf  google::LogMessage::Flush()
    @     0x7f41c032e6ef  google::LogMessageFatal::~LogMessageFatal()
    @     0x7f41bf6693ab  kimera::SemanticLabel2Color::SemanticLabel2Color()
    @     0x7f41bf6a80e6  kimera::getSemanticTsdfIntegratorConfigFromRosParam()
    @     0x7f41c0980035  hydra::incremental::MeshSegmenter::MeshSegmenter()
    @     0x7f41c0966aa8  hydra::incremental::DsgFrontend::startMeshFrontend()
    @     0x7f41c0967c9a  hydra::incremental::DsgFrontend::start()
    @     0x55ec45473f57  main
    @     0x7f41bf7ed083  __libc_start_main
    @     0x55ec4547496e  _start
================================================================================REQUIRED process [hydra_topology_node-10] has died!
process has died [pid 328674, exit code -6, cmd /home/gil/sparoLab/open_source/Graph_slam/devel/lib/hydra_topology/hydra_topology_node --minloglevel=3 -v=0 pointcloud:=/semantic_pointcloud __name:=hydra_topology_node __log:=/home/gil/.ros/log/19f051fe-40d5-11ed-a742-e51d644fcd44/hydra_topology_node-10.log].
log file: /home/gil/.ros/log/19f051fe-40d5-11ed-a742-e51d644fcd44/hydra_topology_node-10*.log
Initiating shutdown!
================================================================================

Could i get some advise for these issues?

Thanks

nathanhhughes commented 1 year ago

@hwan0806 Your issue is unrelated to this one, do you mind opening a separate issue (example title could be "running uHumans2 dataset against other scenes produces no objects") so that other people might find the answer easier? Your comment pulls out the correct args you have to set for the other uhumans2 scenes (typology_config and semantic_map_file)

nathanhhughes commented 1 year ago

@glennliu Thank you for taking the time to fill out this issue (and for your interest in our work). I'm not sure what's causing the crash yet for you. Couple of notes:

To debug, I comment out the launch of hydra_topology/hydra_topology.launch, as I want to confirm the frontend scene graph has been created correctly. Then, the program does not crash anymore and I can see the input sensor data: segmentation images, depth, semantic point cloud. But I cannot see the reconstructed semantic meshes, not to mention object and place markers. An rviz screenshot can be found here,

This makes sense; hydra_topology is responsible for handling the reconstruction of the semantic mesh and the place nodes inside an active spatial window (whereas the frontend in Hydra takes the semantic mesh and place nodes from hydra_topology, extracts the objects, and builds up the scene graph). In other words, hydra_topology is required for building a scene graph for hydra.

Not sure is it because the rostime in the bag is significantly different from my current rostime.

It shouldn't be, we set ROS to use sim time here. Are you running the bag without --clock as an argument? There might be a minor bug in the backend with how we handle timestamps if something isn't publishing to /clock

Couple of questions that would help me reproduce the issue on my side:

glennliu commented 1 year ago

@nathanhhughes Thanks for your time! Here are some dependencies information:

I also play the bag with a --clock option. Still not sure what cause the problem yet.

glennliu commented 1 year ago

I just upgraded my OS to Ubuntu 20.04, and reinstall the ROS Noetic as recommended. pcl 1.10.1 is compiled and installed from the source. So, it is compatible with the default pcl_conversion from ROS.

Then, I run the program roslaunch hydra_dsg_builder uhumans2_incremental_dsg.launch. And it shows, 

process[bag_static_tf_0-2]: started with pid [302432]
process[bag_static_tf_1-3]: started with pid [302433]
process[bag_static_tf_2-4]: started with pid [302444]
process[bag_static_tf_3-5]: started with pid [302449]
process[bag_static_tf_4-6]: started with pid [302451]
process[bag_static_tf_5-7]: started with pid [302462]
process[nodelet_manager-8]: started with pid [302472]
process[cloudify-9]: started with pid [302479]
process[hydra_topology_node-10]: started with pid [302484]
process[incremental_dsg_builder_node-11]: started with pid [302486]
process[pose_graph_publisher_node-12]: started with pid [302491]
process[incremental_dsg_builder_node/pgmo/posegraph_viewer-13]: started with pid [302512]
[ INFO] [1664942779.626716801]: Initializing pose graph visualizer
[ WARN] [1664942779.636948885]: PoseGraphPublisher waiting for subscriber...
double free or corruption (out)
[ INFO] [1664942779.816865297]: Logging backend graph to /home/lch/mit_ws/catkin_new_ws/src/Hydra/hydra_dsg_builder/output/uhumans2/office/backend
[ INFO] [1664942779.823375893]: Logging output to: /home/lch/mit_ws/catkin_new_ws/src/Hydra/hydra_dsg_builder/output/uhumans2/office/pgmo
[ WARN] [1664942779.824905675]: MeshFrontend: Skipping callback registration
[ INFO] [1664942779.825181601]: Initialized MeshFrontend.
[ INFO] [1664942779.825206905]: Logging frontend graph to /home/lch/mit_ws/catkin_new_ws/src/Hydra/hydra_dsg_builder/output/uhumans2/office/frontend
================================================================================REQUIRED process [hydra_topology_node-10] has died!
process has died [pid 302484, exit code -6, cmd /home/lch/mit_ws/catkin_new_ws/devel/lib/hydra_topology/hydra_topology_node --minloglevel=3 -v=0 pointcloud:=/semantic_pointcloud __name:=hydra_topology_node __log:=/home/lch/.ros/log/107f59bc-4463-11ed-93a7-073821ea42ac/hydra_topology_node-10.log].
log file: /home/lch/.ros/log/107f59bc-4463-11ed-93a7-073821ea42ac/hydra_topology_node-10*.log
Initiating shutdown!
================================================================================
[ WARN] [1664942780.041708672]: PoseGraphPublisher waiting for transform: left_cam -> world
[incremental_dsg_builder_node/pgmo/posegraph_viewer-13] killing on exit
[pose_graph_publisher_node-12] killing on exit
[incremental_dsg_builder_node-11] killing on exit
[hydra_topology_node-10] killing on exit
[cloudify-9] killing on exit
[nodelet_manager-8] killing on exit
[bag_static_tf_5-7] killing on exit
[bag_static_tf_4-6] killing on exit
[bag_static_tf_1-3] killing on exit
[bag_static_tf_3-5] killing on exit
[bag_static_tf_2-4] killing on exit
terminate called after throwing an instance of 'boost::wrapexcept<boost::lock_error>'
  what():  boost: mutex lock failed in pthread_mutex_lock: Invalid argument
[bag_static_tf_0-2] killing on exit
*** Aborted at 1664942780 (unix time) try "date -d @1664942780" if you are using GNU date ***
PC: @                0x0 (unknown)
*** SIGSEGV (@0x19) received by PID 302486 (TID 0x7f829c709ec0) from PID 25; stack trace: ***
    @     0x7f82af613090 (unknown)
    @     0x7f82af66a6f0 cfree
    @     0x7f82b09c0d65 hydra::incremental::MeshSegmenter::~MeshSegmenter()
    @     0x7f82b09c0fed hydra::incremental::MeshSegmenter::~MeshSegmenter()
    @     0x7f82b09a4f05 hydra::incremental::DsgFrontend::~DsgFrontend()
    @     0x55e4e4fd6083 main
    @     0x7f82af5f4083 __libc_start_main
    @     0x55e4e4fd6c0e _start
[rosout-1] killing on exit
^C[master] killing on exit

The process crashed before I run any bag of datasets.

nathanhhughes commented 9 months ago

Sorry that we weren't able to figure this out, but it does appear to have been a build issue somewhat specific to your system. I'm going to close this now given that the Hydra code has evolved significantly since this issue was opened, but feel free to reopen if you run into issues with the newest version of Hydra