Loading the models taking a while in demo_handmocap

[khalid-alsaleh-dev]

Thank you so much! Great jop! If i try to predict an image each time it takes about 14 seconds... about 10 seconds to load HandBboxDetector and HandMocap. Can i create a function called 'initialize' and call it only one time...and then i can call the run_hand_mocap function anytime i want. i tried to do that using colab cells...but i faced a problem with the Visualizer getting 'freeglut (foo): failed to open display'. any solutions ,please

[penincillin]

@khalidlionel Thanks for pointing this issue out. Your concern is correct. You can initialize the models once and run the prediction function multiple times. As for the bug, it is related to the OpenGL which might not be able to be used on colab. To resolve use, you can either add xvfb-run as xvfb-run -a python -m demo.demo_frankmocap --input_path ./sample_data/han_short.mp4 --out_dir ./mocap_output or use other screenless renders such as Pytorch-3D --renderer_type pytorch3d or OpenDR --renderer_type opendr.

[khalid-alsaleh-dev]

@penincillin Thank you for your immediate response! but as i told you that i want to do something like this..

cell 1 in colab:

Set Bbox detector

bbox_detector = HandBboxDetector(args.view_type, device)

Set Mocap regressor

hand_mocap = HandMocap(args.checkpoint_hand, args.smpl_dir, device = device)

Set Visualizer

if args.renderer_type in ['pytorch3d', 'opendr']: from renderer.screen_free_visualizer import Visualizer else: from renderer.visualizer import Visualizer visualizer = Visualizer(args.renderer_type)

cell 2 in colab:

run this function anytime you want!

run_hand_mocap(args, bbox_detector, hand_mocap, visualizer)

so I tried not to use the xvfb-run command and used the normal python -m one with --renderer_type pytorch3d but i got this :

qt.qpa.xcb: could not connect to display qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "/usr/local/lib/python3.7/site-packages/cv2/qt/plugins" even though it was found. This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem. Available platform plugins are: xcb, eglfs, minimal, minimalegl, offscreen, vnc.

i hope i made it easy to understand!

[penincillin]

@khalidlionel If you specify --renderer_type pytorch3d then there is no need to add xvfb-run.

[khalid-alsaleh-dev]

@penincillin Yes, i know that but i always get :

qt.qpa.xcb: could not connect to display qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "/usr/local/lib/python3.7/site-packages/cv2/qt/plugins" even though it was found. This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem. Available platform plugins are: xcb, eglfs, minimal, minimalegl, offscreen, vnc.

[penincillin]

@khalidlionel Would you mind tell me the command you use?

[khalid-alsaleh-dev]

@penincillin python -m demo.demo_handmocap --input_path ./sample_data/han_hand_short.mp4 --out_dir ./mocap_output --renderer_type pytorch3d

[penincillin]

@khalidlionel The bug seems to be still related to the display. Would you mind double-checking whether xvfb-run or opengl is used?

[mylee95]

Just in case if anyone else encounter same issue, demo.demo_handmocap needs --no_display option to run with screenless mode, or else it will try to show image on screen and get qt error. https://github.com/facebookresearch/frankmocap/blob/15581a6f005b575256033c2cae184c2b76a0a65b/demo/demo_handmocap.py#L127-L130

Example : python -m demo.demo_handmocap --input_path ./sample_data/han_hand_short.mp4 --out_dir ./mocap_output --renderer_type opendr --no_display