search

lakshjaisinghani / pseudo-lidar-pipeline

A Pseudo Lidar ROS pipeline using BTS monocular depth network.
2 stars 0 forks source link

Question about information required for training other than RGB Depth data #1

Closed mohammad-saber closed 4 years ago

mohammad-saber commented 4 years ago

Thank you very much for sharing your great code. I have a question about training process.

Is there any additional information other than RGB and depth required for training? (Example: camera intrinsic, extrinsic, or other project specific information).

Thank you very much for your help.

lakshjaisinghani commented 4 years ago

Hey Mohammad, That is something I'm working on at the moment actually!

The model being used ideally takes in the focal length as an input parameter while training. But, at the moment I'm not using it at test time due to the following reasons:

  1. The model was pre-trained on the Kitti dataset (rectified data) with the focal length parameter to predict the final depth. final_depth = final_depth * focal.view(-1, 1, 1, 1).float() / 715.0873

  2. While building the pipeline I noticed that the standard deviation between camera focal lengths trained on is very small (same car, same camera, different calibration day).

  3. My test vehicle's camera parameters are totally different.

Hence, the model I found that the model is pretty adept and getting fairly accurate results at test time even if I don't provide a focal length. And the accuracy doesn't need to be great to build the pipeline in the first place (build first, fine-tune later).

Although the model works alright, I'll be changing it to a simpler architecture very soon. The complexity of the network doesn't provide good real-time performance.

I still have quite a bit of work to do on this before it's any good haha!

mohammad-saber commented 4 years ago

May I ask the purpose of using "focal length" in the above formula?

And also, what is the value of "715.0873" here?

lakshjaisinghani commented 4 years ago

if you take a look at the training image file used by the data loader to load kitti images (https://github.com/lakshjaisinghani/pseudo-lidar-pipeline/blob/master/bts_official/train_test_inputs/eigen_train_files_with_gt.txt) you can see that the last term post directory path is the focal parameter. You can see that they vary with each image.

The use of the focal parameter is just to normalize the depth, where the 715.0873 is the average of all focal parameters.