Virtual green screen

[ASchneiderBR]

Hello!

First of all, thanks for all your work here, this is an amazing tool and works really well. Great job!

I was wondering: do you guys have any plans or is it possible to integrate some kind of background remover so we can use it as a virtual green screen?

This would be great to use with OBS with the chroma-key filter to remove the guest background and make it transparent.

Thanks a lot!

[steveseguin]

Definitely something I want to try adding, although there will be some limits to this approach, including the need for a powerful computer, especially if in a group call.

[ASchneiderBR]

Hello @steveseguin ! Thanks for the reply :)

I know this would be very difficult to implement but my first idea was to remove the background directly from the guest hardware, not locally from the director/host machine. Not sure if there's an API for that, like Skype, Teams or Zoom does but only using the browser's camera. There's a whole lot of problems if you take into account that most people join using their mobile phones....

Maybe someday we could have an iOS/Android/Windows app just for that :)

I think a good start would be to add this feature only by reading the camera from a specific URL/parameter so we can do it locally, inside OBS only, just with the ones we need at the moment.

Thanks!

[steveseguin]

I added a 'bokeh' blur effect to https://obs.ninja/?effects=3

If the performance is acceptable to you, I can try to do the same thing but with a green background mask.

(it's not a finished feature; audio is borked and performance needs optimization)

[ASchneiderBR]

I added a 'bokeh' blur effect to https://obs.ninja/?effects=3

If the performance is acceptable to you, I can try to do the same thing but with a green background mask.

(it's not a finished feature; audio is borked and performance needs optimization)

Hello sir! Wow, amazing, thanks for your quick response :)

I'm sorry, but I've invested half an hour before asking for help but I could not make this parameter to work here...

I've created a room, sent the guest invite to my iPhone, shared my camera, copied the view URL and added the ?effects=3 to the parameter but nothing happens... Is that correct or am I doing something wrong here?

I'm really exited about this :) this would be a game changer for me and many friends that I've recommended OBS Ninja, they will for sure stop using NDI and Skype in OBS and switch to your solution.

Thanks again!

[jcalado]

I think this might be only on beta?

So obs.ninja/beta/?effects=3

A sábado, 27/02/2021, 13:37, Antonio Schneider notifications@github.com escreveu:

I added a 'bokeh' blur effect to https://obs.ninja/?effects=3

If the performance is acceptable to you, I can try to do the same thing but with a green background mask.

(it's not a finished feature; audio is borked and performance needs optimization)

Hello sir! Wow, amazing, thanks for your quick response :)

I'm sorry, but I've invested half an hour before asking for help but I could not make this parameter to work here...

I've created a room, sent the guest invite to my iPhone, shared my camera, copied the view URL and added the ?effects=3 to the parameter but nothing happens... Is that correct or am I doing something wrong here?

Thanks again!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/steveseguin/obsninja/issues/750#issuecomment-787073844, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAGR6MWTZMXAZJUF7O33Q3TBDYQBANCNFSM4YGW753Q .

[ASchneiderBR]

I think this might be only on beta? So obs.ninja/beta/?effects=3 A sábado, 27/02/2021, 13:37, Antonio Schneider notifications@github.com escreveu: … I added a 'bokeh' blur effect to https://obs.ninja/?effects=3 If the performance is acceptable to you, I can try to do the same thing but with a green background mask. (it's not a finished feature; audio is borked and performance needs optimization) Hello sir! Wow, amazing, thanks for your quick response :) I'm sorry, but I've invested half an hour before asking for help but I could not make this parameter to work here... I've created a room, sent the guest invite to my iPhone, shared my camera, copied the view URL and added the ?effects=3 to the parameter but nothing happens... Is that correct or am I doing something wrong here? Thanks again! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#750 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAGR6MWTZMXAZJUF7O33Q3TBDYQBANCNFSM4YGW753Q .

Hello! Just tried that and it did not worked as well...

[steveseguin]

Apologies; yeah, it is on beta, not production yet. sorry.

https://obs.ninja/beta/?effects=3

I've only tested it with Chrome PC and macOS currently. It may not work on Mobile, Safari, or Firefox.

It will take 3 to 30 seconds for the "ai model" to download; until it does download, the camera screen might be blank.

If you are using an older computer, it may not work. This requires a GPU I think.

[steveseguin]

[ASchneiderBR]

Apologies; yeah, it is on beta, not production yet. sorry.

https://obs.ninja/beta/?effects=3

I've only tested it with Chrome PC and macOS currently. It may not work on Mobile, Safari, or Firefox.

It will take 3 to 30 seconds for the "ai model" to download; until it does download, the camera screen might be blank.

If you are using an older computer, it may not work. This requires a GPU I think.

Hello!

No worries, thanks for the help. I was able to test using my webcam in Chrome, seems to work fine but the FPS is kinda low.

My PC should not be a problem to run this AI (3900X, 2080ti, 32gb DDR4 3600, Corsair MP600). I've managed to record a video showing its resources being used at 60fps: https://streamable.com/y4tgsu

My first attempt using my iPhone was unsuccessful, I'm afraid. No image was shown.

This was what it shows:

Edit: Well, this is what it shows when I open the camera URL in my PC browser:

It seems that, apart from the stretched image, the BG blur AI model did worked! Ver cool!

Maybe a green screen ai model should be easy on the resources than the blur and also would better visualized if it is doing its job correctly.

Thank you!

[steveseguin]

A green screen would use fewer resources, yes, though the AI model I used still requires a mid-range GPU or so. I can go with a smaller/lighter AI model, but the results will be worse of course.

I did try getting the green screen version done, but I ran into some issues with it, and figured I'd work on it after I got some feedback.

As per frame rate, I think this is addressable, but it will require me to 'buffer' a few frames into a processing queue. I haven't added that code in yet tho. At the moment it does one frame at a time, skipping frames if the previous job was not yet complete. This is where most the low-frame rate issue stems from I think.

[ASchneiderBR]

A green screen would use fewer resources, yes, though the AI model I used still requires a mid-range GPU or so. I can go with a smaller/lighter AI model, but the results will be worse of course.

I did try getting the green screen version done, but I ran into some issues with it, and figured I'd work on it after I got some feedback.

As per frame rate, I think this is addressable, but it will require me to 'buffer' a few frames into a processing queue. I haven't added that code in yet tho. At the moment it does one frame at a time, skipping frames if the previous job was not yet complete. This is where most the low-frame rate issue stems from I think.

Amazing! Please let me know if I can do anything to help, I'll be more than happy to do it.

Thank you very much for your interest and fast response :)

[steveseguin]

I'm running into issues still., but this is sorta what it will look like when done.

[steveseguin]

https://obs.ninja/?effects=4

Green screen added to BETA.

There is still an audio issue and a ghosting of sorts with the masking, but it's smooth.

Also, do not hide or minimize the page; it needs to stay visible for it to work. The electron capture can be used if you need to do otherwise.

[ASchneiderBR]

Hello @steveseguin! Wow, this is evolving really fast hahah Nice!

I've just tested it:

It seems that now my camera is running at ~15fps and the AI green screen at about ~5fps, so there's a lag between my video feed and the green mask. Is that correct, or is it something at my side?

Not sure if it is possible, but I'll suggest it anyways: maybe a warning and a loading for the user to know why there's no image showing, while the AI model downloads would be nice in the future.

The resources used seems to be fine, my GPU is running at a very low clock using a reasonable to low amount of resources, I would say, seems very nice to me.

The AI Model seems OK at this point, I know that using a better one would be more resources intensive but maybe an option to chose from? I love the way Teams/Skype/Zoom removes the background, they use a very nice model with fathered masks, very accurate.

Thank you very much, again.

[steveseguin]

&effects=5 is the biggest model I could find, at like 90-mb in size. It takes a while to load and isn't much better.

There is a lag between the mask being applied and the video. I'm struggling to find a good solution to that.

I can add feathering. I found that once I bring the image into OBS though, and chroma-key out the green, the green feathering caused a green glow around the edges to remain.

I find feathering makes sense if I were to add a background image directly, as I would have transparency then, but for the purpose of a green screen, feathering seems to not be helpful. I could be wrong though

[ASchneiderBR]

Oh, makes sense... Here's a test - https://streamable.com/4dr0o7

I really hope you can find something that works, specially with mobile phones that I believe is the best way to send a remote camera to OBS.

Thank you very much Steve.

[kar-re]

One idea would to apply the AI model on the incoming video stream, I'm unsure whether this would create any issues in terms of 'loop-back' canvases' needed to apply the effect but that would solve the mobile vs desktop issues @tatunts mentioned! :)

I.e. pushing the camera using ?push=xxxxx and viewing using ?effects=3&view=xxxxx

As always, thank you very much Steve. You're a star.

[steveseguin]

bodypix isn't seemingly compatible with iOS, so I'm not going to put too much effort into trying to get it working there for now. https://github.com/tensorflow/tfjs/issues/3624

things are working with android chrome just fine though, minus the lag/performance aspect, but that's a given.

I'll need eventually look for a better model or make my own specific to the task at hand. Not too happy with the models currently offered..

[affinage-digital]

Hello Steve!

I think better web-based solution google meet. I have spent a lot of time, and found this https://ai.googleblog.com/2020/10/background-features-in-google-meet.html?m=1

They used solution MediaPipe https://google.github.io/mediapipe/solutions/solutions Only js based https://google.github.io/mediapipe/getting_started/javascript

I hope this will be useful for you and your great work! If you are interested I will be happy to help you to integrate into you solution. I'm sure it will be awesome.

[steveseguin]

Thank you kindly. I will explore. Performance has been a blocker so far.

[steveseguin]

An improved version has been added to alpha. https://obs.ninja/alpha

You can find the options in the settings or by add &effects to the URL. It's still buggy and it needs WASM + SIMD support enabled in the chrome://flags settings, but it should do what was requested.

[ASchneiderBR]

Hello Steve! I was wondering: is it possible to add a view parameter so we can enable this effect locally from a remote video feed? This would help a lot to use this effect on devices, unable to do it like the iPhone.

This project seems really nice but it is a totally different approach: https://github.com/royshil/obs-backgroundremoval

I think that might be some limitations because of the OBS native browser, I think you need to set some flags in Chrome to make this effect work correctly, right?

Thanks!

[GladstoneFilho]

Hello Steve! I was wondering: is it possible to add a view parameter so we can enable this effect locally from a remote video feed? This would help a lot to use this effect on devices, unable to do it like the iPhone.

This project seems really nice but it is a totally different approach: https://github.com/royshil/obs-backgroundremoval

I think that might be some limitations because of the OBS native browser, I think you need to set some flags in Chrome to make this effect work correctly, right?

Thanks!

Hi Did you find any mobile alternative for OBS's camera with green background?

I'm trying using https://obs.ninja/?mirror&effects=4 on iOS but still doesn't work =(

[ASchneiderBR]

Hello Steve! I was wondering: is it possible to add a view parameter so we can enable this effect locally from a remote video feed? This would help a lot to use this effect on devices, unable to do it like the iPhone. This project seems really nice but it is a totally different approach: https://github.com/royshil/obs-backgroundremoval I think that might be some limitations because of the OBS native browser, I think you need to set some flags in Chrome to make this effect work correctly, right? Thanks!

Hi Did you find any mobile alternative for OBS's camera with green background?

I'm trying using https://obs.ninja/?mirror&effects=4 on iOS but still doesn't work =(

Hello, no. I'm still looking for a solution for that...

[steveseguin]

I'll spend more time on the iOS version, although my current goal was to add the option to do the green screen on the viewers/director's side instead. Trying to get it working with Safari so far has been challenging.

[ASchneiderBR]

green screen on the viewers/director's side

Hello Steve! That's perfect, I also think this is the way... Thanks.