search

intel / openvino-plugins-for-obs-studio

Apache License 2.0
53 stars 7 forks source link

Migrate the video features (background blur; face mesh, smart framing) to gstreamer #3

Closed andyyeh75 closed 1 year ago

andyyeh75 commented 1 year ago

I am trying to migrate the openvino video features (background blur; face mesh, smart framing) to gstreamer. However I know there are the differences between OBS and gstreamer. How could I start this work?

(I am referring this repo https://github.com/dlstreamer/dlstreamer as the sample to run NN models (IR or ONYX) formats on a dlstreamer gstreamer plugin. )

RyanMetcalfeInt8 commented 1 year ago

Hi @andyyeh75, it might be somewhat complicated to port these features to GStreamer. It's been a while since I've worked with DLStreamer -- it might help you to run the inference portion, but then I think you would need to develop a custom plugin, or maybe use something like gvapython to grab the inference results from the GstBuffer's and then perform the video manipulation (background blur/replacement for background removal, crop/scale for smart framing, etc.).

RyanMetcalfeInt8 commented 1 year ago

And since DLStreamer ideally wants to perform pre/post processing on your behalf, I'm not sure if it would be able to understand the specifics of those routines for the mediapipe models that we are using. For example, take a look at https://github.com/intel/openvino-plugins-for-obs-studio/blob/main/ovmediapipe/src/face_detection.cpp, this source file includes the correct logic for taking an input frame, applying pre-processing, and also the method to take inference results & convert them to bounding boxes.

RyanMetcalfeInt8 commented 1 year ago

The easiest route in my opinion, if you still want to proceed, is to develop a standalone set of GStreamer plugins written in C++, and make the exact same function calls that we do in the OBS plugin itself. They would both have a similar start / end point, being an input frame passed in, and output frame set as output. You would just need to adapt it to use GstBuffer's (along with the caps negotiation piece, etc.)

andyyeh75 commented 1 year ago

Thanks @RyanMetcalfeInt8 (sorry for my reply late) It is more complicate than I thought. I will study the gstreamer plugin to write one for accessing the video manipulation features. closed it.