search

facebookresearch / pytorchvideo

A deep learning library for video understanding research.
https://pytorchvideo.org/
Apache License 2.0
3.28k stars 406 forks source link

Webcam implementation #42

Open Mathuran-Mang opened 3 years ago

Mathuran-Mang commented 3 years ago

I was wondering if there was a way to do action recognition with pytorchvideo using a live webcam

tullie commented 3 years ago

It definitely would be possible. We've shown greater than realtime performance for the X3D accelerated models so they might fit your use case well. The only thing that may be an issue is how well the pretrained models generalize to the webcam footage you're using. Ideally you'd be able to fine-tune the models on a dataset of the webcam footage.

Mathuran-Mang commented 3 years ago

@tullie isn't the X3D accelerated models only used for mobile devices? Or is there a way to run them on my webcam through my pc?

tullie commented 3 years ago

@liyilui will the accelerated models work on a computer cpu as well? Or is it coupled tightly to the mobile architecture?

liyilui commented 3 years ago

You can run the X3D model on a computer CPU also, but the convert_to_deployable_form may not bring latency reduction for desktop x86 CPU (it should benefit laptop with ARM CPU though, for example some models of Microsoft Surface). Referring to the tutorial , you can compare the latency with or without convert_to_deployable_form, and see which one is better for your desktop CPU.

Mathuran-Mang commented 3 years ago

@liyilui I am a little confused on how I can deploy this model on to my computer CPU using a webcam to do human action recognition. Is there any way you can give me instructions on how to do this? I would really appreciate it.

liyilui commented 3 years ago

@Mathuran-Mang There is one example of running demo on webcam for human action recognition from slowfast codebase , I believe that would be a good starting point .

Mathuran-Mang commented 3 years ago

Thank you will look into it

Mathuran-Mang commented 3 years ago

@Mathuran-Mang There is one example of running demo on webcam for human action recognition from slowfast codebase , I believe that would be a good starting point .

@liyilui I am having trouble running the webcam demo, do I just have to call the run_demo function from the code that you gave me the link to? run_demo(cfg, frame_provider) I try replacing frame_provider with the video capture function for opencv but that does not work. Would you mind helping me with this?

liyilui commented 3 years ago

@Mathuran-Mang There is one example of running demo on webcam for human action recognition from slowfast codebase , I believe that would be a good starting point .

@liyilui I am having trouble running the webcam demo, do I just have to call the run_demo function from the code that you gave me the link to? run_demo(cfg, frame_provider) I try replacing frame_provider with the video capture function for opencv but that does not work. Would you mind helping me with this?

@haooooooqi Could you take a look (or direct to right person) about this (regarding demo using webcam in slowfast, or make a similar version here)?