search

TempleRAIL / drl_vo_nav

[T-RO 2023] DRL-VO: Learning to Navigate Through Crowded Dynamic Scenes Using Velocity Obstacles
https://doi.org/10.1109/TRO.2023.3257549
GNU General Public License v3.0
113 stars 7 forks source link

source of pedestrian maps #4

Closed ConnerQiu closed 1 year ago

ConnerQiu commented 1 year ago

Hi zhanteng, I would like to ask a question related to the pedestrian maps:

  1. In the paper, the pedestrain maps is obtained from image from ZED camera with yolov3 and MHT algorithm. While when I check the rqt_graph after runing the training algorithm, I notice that the “track_ped” topic is originally traced back to the “pedsim_simulator” node, and the image from camera “/zed/…/image_raw_color” topic is subscribed by no nodes. So, it looks like the pedestrian maps is calculated while the “pedsim_simulator” control the movement of pedestrians, not from the image. In addition, I try to locate the yolo algorithm in the code, but did not find one. Is my understanding right, or do I just came across some bugs with my code?
  2. If pedestrain maps is from “pedsim_simulator”, how does it compute the relative location of pedestrian and robot. Since it does not subscribe the information about robots, this seems impossible. Also, when I check the image in rivz, I cannot see the bounding box of pedestrian, so I guess something has go wrong inside the code I deployed.
zzuxzt commented 1 year ago

Hi Ronghe Qiu, here are my answers:

  1. Sorry for the inconvenience, the current open-source code does not include the Yolo detector and MHT tracker, and the pedestrian information comes directly from the pedsim_simulator. We did not open source the Yolo detector and MHT tracker code because they are more for the hardware side and make the simulator run slower.
  2. The pedestrian information from the simulator is actually the ground truth information. I have already transferred it to the robot's local coordinate frame (i.e. "track_ped" topic). You also can't see the bounding boxes in the Rviz with the current setting because the pedestrian information comes directly from the simulator (skipping the detector and tracker pipeline).
ConnerQiu commented 1 year ago

Thanks, Zhanteng.Do you also use the ground truth information of pedestrian during your training in gazebo.Only in sim-to-real experiment, since such oracle information is not available, the Yolo detector and MHT tracker is used. Such setting is manily to accelerate the training in simulator since with them the training is already quite slow. Is my understanding right? I think it is quite reasonable. The gap between ground truth and output from MHT tracker can be tolerate during the simulation training.

Can I also get the code about process images with Yolo detector and MHT tracker and then output the pedestrain map if it is convenient. That would help me a lot.

zzuxzt commented 1 year ago

As shown in my paper, I used the YOLO & MHT pipeline to train the policy (trying to minimize the sim-to-real gap). However, since the preprocessing pedestrian map representation already bridges the sim-to-real gap well, using ground truth data is acceptable and speeds up training. If you want to further close the sim-to-real gap, you can add some noise to the ground truth data.

Sorry, we currently have no plans to open source the code for these hardware implementations, as they require some commercial license packages, which is a bit complicated. We will think about open sourcing them when we clean up these constraints and update the code. Thanks for your understanding.