search

ROBOTIS-JAPAN-GIT / turtlebot3_slam_3d

Turtlebot3 3D-SLAM demo using RTAB-Map with Jetson TX2 and ZED Mini
182 stars 62 forks source link

Bring up zed camera #17

Open darissa opened 2 years ago

darissa commented 2 years ago

Hi @Affonso-Gui. I successfully run this code. Now, I wanna change to my environment which I am using ZED2 camera only with yolov4 object detection. So first step I try to change the input to ZED camera, live input. FYI, I have changed several things:

  1. changes node to /zed_node/left/image_rect_color
  2. changes node to /zed_node/depth/depth_registered
  3. changes node to /zed_node/left/camera_info
  4. changes node to /zed_node/depth/camera_info
  5. changes node to /zed_node/odom
  6. changes node to /zed_node/point_cloud/fused_cloud_registered
  7. comment out turtlebot3_robot_launch and publish_model.launch in turtlebot3_zed_bringup
  8. comment out visual_odom line in turtlebot3_slam_3d.launch since already set to false
  9. add nodelet depthimage_to_laserscan in rtabmap.launch
  10. merge YoloObjectDetector.cpp file to this github https://github.com/AkellaSummerResearch/darknet_ros.git, because I already test, able to run live zed for yolo object detection in ROS.
  11. add these lines in darknet_config.yaml

subscribers:

camera_reading: topic: /zed_node/left/image_rect_color queue_size: 1

image_view:

enable_opencv: true wait_key_delay: 1 enable_console_output: true use_darknet: true

  1. add these lines in yolov3.yaml

image_view:

enable_opencv: true wait_key_delay: 1 enable_console_output: true use_darknet: true

When I run roslaunch turtlebot3_slam_3d turtlebot3_slam_3d.launch, I got warning as below, and YOLO V3 window not pop-up.

/rtabmap/rtabmap: Did not receive data since 5 seconds! Make sure the input topics are published ("$ rostopic hz my_topic") and the timestamps in their header are set. If topics are coming from different computers, make sure the clocks of the computers are synchronized ("ntpdate"). If topics are not published at the same rate, you could increase "queue_size" parameter (current=600). /rtabmap/rtabmap subscribed to (approx sync): /zed_node/odom \ /zed_node/left/image_rect_color \ /zed_node/depth/depth_registered \ /zed_node/left/camera_info \ /rtabmap/scan

Also, I checked all changed node are working except /zed_node/point_cloud/fused_cloud_registered.

Already stuck with this thing about a week. Hope you can help. Thank you.

darissa commented 2 years ago

UPDATE:

Now, able to view YOLO V3 window and object detection works well. However, I got different error "segmented cloud size is zero" and map is not received in rviz. When I checked rostopic hz /map or any related topic to rtabmap, "no new messages" output.

Affonso-Gui commented 2 years ago

Hello,

This repository is quite old and active development has been dropped a few years ago when I changed position. In order to keep it running most of the external code has been pinpointed to a certain version (https://github.com/ROBOTIS-JAPAN-GIT/turtlebot3_slam_3d/blob/master/.rosinstall). A lot has changed since then, unfortunately mostly without backward compatibility, as you have been experiencing.

This is probably a good time to update some of the code, do you think you can open a PR with these changes? Having access to your running code (and maybe a small bag file) would also help me a lot in debugging this problem!

darissa commented 2 years ago

@Affonso-Gui Sorry a bit late. I am new to github. So I just open a PR. You can checkout. I will do the same for darknet_ros.git. Thank you so much for your support. I think this project is amazing, I actually seeking this kind of output quite some time. That's why I really try hard on your github. Tq Tq.

Affonso-Gui commented 2 years ago

Hello,

I couldn't find the Pull Request or any changes in your working branch... Could you check it once again and post the link to the new PR here?

darissa commented 2 years ago

Hi,

https://github.com/darissa/turtlebot3_slam_3d/tree/new_branch

darissa commented 2 years ago

This one for darknet_ros. I didn't change any for zed_wrapper.

https://github.com/darissa/darknet_ros_zed/tree/new_branch

Thank you

darissa commented 2 years ago

Hi,

I successfully generate the map input from live zed camera and svo file. However, i got warning:

"[ WARN] (2022-04-14 13:23:10.245) SensorData.cpp:718::uncompressDataConst() Requested laser scan data, but the sensor data (193) doesn't have laser scan."

and..even though yolo works well, i mean, they have the detection, but when run detection_collector.py, i got empty file.

WhatsApp Image 2022-04-14 at 1 33 20 PM

Affonso-Gui commented 2 years ago

even though yolo works well, i mean, they have the detection, but when run detection_collector.py, i got empty file.

Something seems to be going wrong in the detection pipeline (which transforms 2d coordinates from the yolo detection image into 3d map coordinates). https://github.com/darissa/turtlebot3_slam_3d/blob/new_branch/launch/darknet.launch#L39-L60

Which of the following topics are you receiving? (rostopic hz)

/darknet_ros/detection_image
/darknet_ros/label_image
/darknet_ros/cluster_points
/cluster_decomposer/centroid_pose_array
Affonso-Gui commented 2 years ago

"[ WARN] (2022-04-14 13:23:10.245) SensorData.cpp:718::uncompressDataConst() Requested laser scan data, but the sensor data (193) doesn't have laser scan."

This warning seems a bit odd considering that you are not setting the subscribe_scan right now https://github.com/darissa/turtlebot3_slam_3d/blob/new_branch/launch/rtabmap.launch#L36

Have this occurred only once or several times? It might be simply your cache kicking in; try to explicitly set subscribe_scan to false instead of commenting it out.

darissa commented 2 years ago

Yes, you right. The collector doesn't work. Receiving these three only:

/darknet_ros/detection_image /darknet_ros/label_image /darknet_ros/cluster_points

Didn't received rostopic hz /cluster_decomposer/centroid_pose_array. I have checked, the point cloud topic received no new message. I subscribed to the topic zed_node/point_cloud/fused_cloud_registered

Yes, I already subscribe_scan to false. It occurred several times. I have checked the laserscan works but the warning still appear. I am using depthimage_to_laserscan node.

darissa commented 2 years ago

Thanks to you, now the collector is working after changing the topic to zed_node/point_cloud/cloud_registered.

Then, I run the publisher, the label cannot publish. Now the warning 'segmented cloud size is zero' and 'Requested laser scan data, but the sensor data (989) doesn't have laser scan' still have. And got another warning in rviz and the image cannot display. Also, as you can see in the figure, I think the z-axis is not at 0 right. Where to fixed this parameter?

WhatsApp Image 2022-04-16 at 7 00 51 AM

Affonso-Gui commented 2 years ago

Just to make sure, can you also see /zed_node/point_cloud/fused_cloud_registered ? What is the output from the following?

rostopic info /darknet_ros/cluster_points
rostopic info /zed_node/point_cloud/fused_cloud_registered

How many Hertz on each topic?

If everything seems fine and connected, it might be a synchronization issue. Even though we are setting approximate_sync to true, the default queue_size of 100 might still not be enough.

https://jsk-docs.readthedocs.io/projects/jsk_recognition/en/latest/jsk_pcl_ros/nodes/cluster_point_indices_decomposer.html

The best would be to have the same header in the yolo input and output image, and then set approximate_sync to false. This might have been covered here: https://github.com/leggedrobotics/darknet_ros/issues/104 https://github.com/leggedrobotics/darknet_ros/pull/113 https://github.com/leggedrobotics/darknet_ros/issues/104#issuecomment-468262691

If the problem persists, uploading a short bag file in your fork would help me to dive deeper into the bug!

Affonso-Gui commented 2 years ago

Oh, nevermind, glad you fixed it! Can you also update your changes to the fork?

darissa commented 2 years ago

These samples are detected image. Is it weird to have -ve value? Is that why the cloud is in the middle of the plane of z-axis?

Found a tvmonitor at [-2.1779866218566895, -11.493218421936035, 0.6778092980384827] Found a chair at [-3.11004638671875, -11.725285530090332, 0.34713223576545715] Found a tvmonitor at [-2.2595298290252686, -8.947291374206543, 0.5882754921913147] Found a person at [-2.6636433601379395, -9.517132759094238, 0.6968424916267395] Found a chair at [-3.9244256019592285, -10.729484558105469, -0.09537818282842636]

darissa commented 2 years ago

Oh, nevermind, glad you fixed it! Can you also update your changes to the fork?

Alright will do.

Affonso-Gui commented 2 years ago

the label cannot publish

What are the contents of the detections_raw.db and detections_dbscan.db generated by the collector script?

darissa commented 2 years ago

These are the contents of the detection_dbscan.db

chair:

darissa commented 2 years ago

Hi @Affonso-Gui, already update my repo.

https://github.com/darissa/turtlebot3_slam_3d/tree/new_branch/launch.

darissa commented 2 years ago

@Affonso-Gui I just want to clarify this thing. I am using zed only. Then using depthtolaserscan node to have laser scan topic. But i got error 'Requested laser scan data, but the sensor data (989) doesn't have laser scan'. Is it because of this laser scan, it can't publish the label?

Affonso-Gui commented 2 years ago

@darissa Sorry for the delay, I have been busy lately but will take a look as soon as possible.

ajrillah commented 2 months ago

sorry, which file should i modify from zed wrapper for custom detector? because i am using ros2 zed wrapper and i want to use yolov8.