search

Crowsinc / LiveVisionKit

LiveVisionKit brings the powers of computer vision and image processing to OBS Studio; implementing state of the art filters such as image enhancement and real-time video stabilization.
GNU General Public License v3.0
153 stars 14 forks source link

Real-Time Video Stabilization for game sources #3

Closed ASchneiderBR closed 2 years ago

ASchneiderBR commented 2 years ago

What is the requested feature? Hello! This plug-in is simply amazing, thanks for creating it. Do you think it would be possible to make Real-Time Video Stabilization work with a game source? Any other kind a part from media sources, really... I think that would be extremely useful.

Why do you want this feature? This would help a lot while streaming VR first person view content, because it shakes a lot and it is not very pleasant for the viewer.

Possible Use Cases VR content, stabilizing all kinds of footages from other source types .

Thank you very much!

ASchneiderBR commented 2 years ago

Here's an example: https://streamable.com/12p80w

Crowsinc commented 2 years ago

The video stabilizer and the other media filters need access to raw frame data which OBS only provides to video filters. I'm currently experimenting with ways of bypassing this limitation. If it works out, then there will be a future update that will will let you run any of the LiveVisionKit video filters as effect filters. The video stabilizer also needs to delay audio which cannot be done with an effect filter. I am aware of a potential work around for this which is a little unpleasant but should let you run the stabilizer on a game when the update comes.

I can't give you an ETA at this point, but it is a priority on my list.

ASchneiderBR commented 2 years ago

Hello. That makes a lot of sense. Looking forward to that! Thanks for the great work here.

Crowsinc commented 2 years ago

Hi, some form of this is expected to be released in the next feature update.

Crowsinc commented 2 years ago

This has now been released as part of LiveVisionKit 1.2.0 (currently Windows only). You will need to manually add an audio delay to match the stabilizers video delay. Check the wiki page for more information.

ASchneiderBR commented 2 years ago

Amazing! Thank you very much. I’ll give it a try and let you know ASAP.

ASchneiderBR commented 2 years ago

Hello @Crowsinc! Just test and it works great, thank you very much sir. I was wondering: do you know if it would be possible to have some kind of per object motion blur in real time? Something like this would be just awesome to have but also very resource intensive as well haha Thanks.

Crowsinc commented 2 years ago

It's probably possible to implement an approximation of such a filter, but not to the same level as the ReelSmart Motion Blur you mentioned. They use per-pixel tracking, which is too computationally expensive to run at over 60FPS. And it tends to produce bad tracking as objects move around and reveal/cover pixels, which is another problem that needs to be solved computationally for live video because you can't 'undo' bad tracking.

What's the application for this?

ASchneiderBR commented 2 years ago

Thanks for answering! This is purely aesthetic. The image resulting looks a lot more realistic, fluid and most people just loves it when they see it. Most games (especially VR) doesn't have this effect so having it rendered in real time would be awesome.

These are some tests I've done:

https://streamable.com/9rmrh5 https://streamable.com/g4rjrs https://streamable.com/m2z6gc

Thanks!