search

StanfordVL / JRMOT_ROS

Source code for JRMOT: A Real-Time 3D Multi-Object Tracker and a New Large-Scale Dataset
MIT License
144 stars 35 forks source link

Transformation lidar to rgb #11

Closed brudermueller closed 3 years ago

brudermueller commented 3 years ago

Thank you for your great work and this impressive dataset. I was having a closer look at this repo in order to understand how you are using the new JRDB for your purpose of tracking. I want to use it in the context of 3D people detection and I am currently wondering which transformations to apply.

I am training my detection network in the Velodyne coordinate system (i.e. x: forward, y:left, z:up) but something still seems odd, when visualizing the merged point clouds (upper and lower lidar) and the 3D bounding box labels. I have first tried transforming both point clouds to the base coordinate system, as well as the center coordinates (transformations taken from here). Alternatively, I tried to translate only the point cloud from the lower velodyne coordinate system into the upper velodyne coordinate system, since the bounding boxes have been annotated with reference to the upper velodyne coordinate system. Then I undo the centering of the center annotations on the RGB camera by translating them back up in the upper velodyne coordinate system. In my understanding both approaches should lead to the same result? But also in the visualization the point cloud appears to be a bit spherical.

Do you have any recommendation on which transformations to apply and to which coordinate system to transform all the point clouds and annotations? I also upload my code for the transformations here: jrdb_transform.py.zip

Moreover, I am not sure, if there might be a bug in the function _ move_lidar_to_cameraframe in the calibration source code (see. https://github.com/StanfordVL/JRMOT_ROS/blob/2b72612d73b212f894ff1730f20926183f33217c/src/calibration.py#L435). You subtract the translation vector (loaded from the calibration.yaml) from the points but this reverses the effect of the translation vectors as defined in the calibration.yaml (lidar_upper_to_rgb: translation: [0, 0, -0.33529]). I think either in the calibration.yaml or in the function the minus should be inverted, since the camera sits in between the upper and lower lidar sensor as far as I understood?

Thanks in advance for your advice and help! I would highly appreciate your input.

mvpatel2000 commented 3 years ago

Hi @brudermueller, we'd be happy to help! It seems like your questions are more about the JRDB dataset though. Could you please send this as an email to jrdb@cs.stanford.edu? This will let the full team see your request and have higher visibility.

yustaub commented 3 years ago

hi, brudermueller. How do you train jrdb dataset? I also want to process it but I have no idea how to handle it because I have only process kitti dataset.

Pradur241 commented 8 months ago

Same question about training jrdb dataset!! I tried to convert jrdb to kitti's training format, but something didn't seem right in the point cloud processing part. Do you have any idea about it? Help, please!! @brudermueller @mvpatel2000