Inferencing in the wild with Detectron2

[FamousDirector]

With the release of Detectron2 it would be awesome to see inferencing in the wild be reworked to integrate with Detectron2. Is there any plans for this?

[doubleZ222]

So,by now the project isn't supported by Detectron2 ?

[darkAlert]

Here is a script that prepares data for VideoPose3D (in .npz format) using Detectron2: https://github.com/darkAlert/VideoPose3d_with_Detectron2/blob/master/detectron_pose_predictor.py

After executing this script, you can go to step 5: https://github.com/facebookresearch/VideoPose3D/blob/master/INFERENCE.md#step-5-rendering-a-custom-video-and-exporting-coordinates

[immkapoor]

@darkAlert Hi, I was trying to run your code. Though I am able to use detectron2 for prediction in my system which has only CPU. Though running your code throws an assertion error of NVIDIA drivers. Do you know any point which I am missing here?

[darkAlert]

@immkapoor I have just added "cpu mode" to the code. Now you just need to set "cuda=False" here: https://github.com/darkAlert/VideoPose3d_with_Detectron2/blob/master/detectron_pose_predictor.py#L156

[immkapoor]

@darkAlert Yes, it's working. Thank you

[FRunyang]

I run the script, there is no indication of progress, but the cpu is using, about 90%. Is it normal?

[FRunyang]

@immkapoor I have just added "cpu mode" to the code. Now you just need to set "cuda=False" here: https://github.com/darkAlert/VideoPose3d_with_Detectron2/blob/master/detectron_pose_predictor.py#L156

Hello,I run the script, there is no indication of progress, but the cpu is using, about 90%. Is it normal?

[FRunyang]

@immkapoor I have just added "cpu mode" to the code. Now you just need to set "cuda=False" here: https://github.com/darkAlert/VideoPose3d_with_Detectron2/blob/master/detectron_pose_predictor.py#L156

and the model （model_final_5ad38f.pkl）I download in https://dl.fbaipublicfiles.com/detectron/37698009/12_2017_baselines/e2e_keypoint_rcnn_R-101-FPN_s1x.yaml.08_45_57.YkrJgP6O/output/train/keypoints_coco_2014_train:keypoints_coco_2014_valminusminival/generalized_rcnn/model_final.pkl ,
Is it correct?

[FRunyang]

@immkapoor I have just added "cpu mode" to the code. Now you just need to set "cuda=False" here: https://github.com/darkAlert/VideoPose3d_with_Detectron2/blob/master/detectron_pose_predictor.py#L156

where can download model_final_5ad38f.pkl ？

[darkAlert]

@huaqi19 you can download it from here https://dl.fbaipublicfiles.com/detectron2/COCO-Keypoints/keypoint_rcnn_X_101_32x8d_FPN_3x/139686956/model_final_5ad38f.pkl

There are also other models here https://github.com/facebookresearch/detectron2/blob/master/MODEL_ZOO.md

[slava-smirnov]

@darkAlert I can confirm cuda=false and all of the above works flawlessly

[fabiopoiesi]

Thanks for the code.

I noticed that some of the estimated 3D joints are different from those in 2D. Especially the top part of the head and the torso (see screeshot). I am looking to back project the 3D pose on the image and estimate the re-projection error. But because they do not correspond, I am actually stuck!

[immkapoor]

@fabiopoiesi can you please explain what do you mean by 3D joints are different from 2D?

The 3D retrieved pose here is correct, The red part is in front and black at back just like in your image. To back project you need to either use a learning method or have a setup with stereo vision.

[fabiopoiesi]

For example: The torso is a line composed of 3 keypoints in the 3D version (they follow the spine), whereas is a rectangle composed of 4 keypoints in the 2D version (2 points on the shoulders two points on the hips)

The head is modelled with 3 keypoints in the 3D version (neck + face + head), whereas is modelled with 5 keypoints in the 2D version (2 on the ears, 2 on the eyes and 1 on the nose)