search

LeCAR-Lab / ABS

[RSS 2024] Agile But Safe: Learning Collision-Free High-Speed Legged Locomotion
https://agile-but-safe.github.io/
254 stars 23 forks source link

Question regarding the raycasting network #3

Closed Zhefan-Xu closed 6 months ago

Zhefan-Xu commented 7 months ago

Thanks for the great work. I have some questions about the ray prediction network. In the paper, it is mentioned that the robot is equipped with a depth camera. Why it is necessary to train a ray prediction netowork instead of directly projecting depth image pixels using camera intrinsics into 3D space to obtain the ray?

zita-ch commented 6 months ago

There are objects with holes together with image noises. Tuning heuristics can be tricky. Training an NN brings data-driven robustness.

Zhefan-Xu commented 6 months ago

Thanks for your explaination. I understanded your motivation. I think this is an interesting discussion, so I have some further questions:

  1. I agree with the robustness portion w.r.t to noises and missing values. However, if using depth image directly to output 3D points, does that mean the network only works with the trained camera intrinsics?
  2. How does the trained ray compare to the ray projected by the camera intrinsics in terms of accuracy? (forgive me if I missed that part in the paper)
  3. Will training a precdictor to the image space (e.g. a image-to-image VAE) work better to deal with noises/missing values? (since it does not depend on camera specs)
  4. The figure in the paper shows sparse ray casting. Would it be easier to just use a 2D LiDAR? Even 2D LiDAR seems more robust and accurate compare to the depth camera.

Apologize for putting up lots of questions. I really appreaciate your reply and it would be very beneficial for my understanding!

zita-ch commented 6 months ago
  1. Yes. One NN for one cam setting.

2/3. Unfortunately we cannot offer a nicely quantified comparison. It depends on your use case. And this is not our major contribution. We encourage you to further study it if you are interested.

  1. This is due to engineering concern. There is no 2d lidar that is light for go1 to carry while reaching 40Hz scanning rate.
Zhefan-Xu commented 6 months ago

Thanks for the answer. Again, thanks for bring this great work to the community!