Run on raw live video

[zillur-av]

I downloaded the pre-trained weighs. I would like to use the kitti weights to run on my raw video/webcam and get the output with 3d box and bird's eye view. How can I do that? Also, how do I include the my extrinsic and intrinsic camera calibration parameters?

[abhi1kumar]

Hi @Zillurcuet

I downloaded the pre-trained weighs. I would like to use the kitti weights to run on my raw video/webcam and get the output with 3d box and bird's eye view. How can I do that?

I have not tried this, but here are the steps to do it for other images:

• The current codebase needs inputs in the KITTI format. Therefore, prepare the dataset in the following format:

├── data
│      ├── KITTI
│      │      ├── ImageSets
│      │      ├── kitti_split1
│      │      └── testing
│      │            ├── calib
│      │            └── image_2
│      │

• Change the paths in these lines of the KITTI dataloader.

• Then, make inference over the trained KITTI model to generate the labels using the following command from the inference script

  CUDA_VISIBLE_DEVICES=0 python -u tools/train_val.py --config=experiments/run_250.yaml\
  --resume_model output/run_250/checkpoints/checkpoint_epoch_140.pth -e

  That will generate labels in the output folder.

• Finally, use the plotting function

  plot/plot_qualitative_output.py --dataset kitti --folder output/run_250/results_folder/data

You need to chain the above four steps in a loop for a raw video. Please feel free to contribute the demo code to our repo by opening a pull request.

Note: Mono3D models (unlike 2D detection) are sensitive to the dataset and do not perform well if tested on another dataset. See Fig. 14 and Tab 6 of our paper

Also, how do I include the my extrinsic and intrinsic camera calibration parameters?

That goes inside the text files of the calib folder. P2 is the 3x4 camera calibration matrix and is the product of camera intrinsics and extrinsics matrix.

P2: 7.070493000000e+02 0.000000000000e+00 6.040814000000e+02 4.575831000000e+01 0.000000000000e+00 7.070493000000e+02 1.805066000000e+02 -3.454157000000e-01 0.000000000000e+00 0.000000000000e+00 1.000000000000e+00 4.981016000000e-03