search

nexmo-community / build-a-thing-video

A project being built live every two weeks at twitch.tv/vonagedevs
MIT License
3 stars 9 forks source link

Greenscreen background image #13

Closed RascalTwo closed 3 years ago

RascalTwo commented 4 years ago

If bug fixes or new features

You can only publish a raw video source, closes #5.

You can now publish a video source with colors chroma keyed and replaced with a background image.

The user can upload a image of their choice, and once uploaded, customize the X/Y position, width, and height of the image.

To assist with positioning, a Overlay option is provided to draw the image on top of the video.

Additionally, the user may select a low and high colors as their chroma key color if they wish to use a non-green color.

Although it's not the most user friendly due to using the native color picker, which uses various color representations - HSV, RGB, etc - while the actual color representation used for calculations is HSL.

No

In order to have better performance, the image manipulation is done in a Worker.

I attempted to make it reusable by exposing a getUserStream method, so the user can be given another chance to provide a video source by calling this method again.

I couldn't come up with a simple way to directly return the native stream when no background image is in use, but have it replaceable with the canvas one when the background image is in use.

Therefore even when there is no background image, the video is still being passed through a canvas.

I avoided actual styling of the form aside from making it collapsible with the Open/Close button.

While requestAnimationFrame() is used, it's limited to the reported frame rate of the user input video.

I had intended to pass the ImageData by reference to the Worker, but I ran out of time - doing this I imagine would further increase performance, with only modifications to sending/receiving portions of the code being required.

My attempt to use the implementation described here did not have positive results in what testing I did - the inability to use a Worker resulted in decreased performance, and this was before I was even able to implement all the existing customization features - you can see it here if you're interested.

netlify[bot] commented 4 years ago

Deploy request for build-a-thing-video pending review.

Review with commit 9f823a8444346fb6cdeebc29243d6d2923201b14

https://app.netlify.com/sites/build-a-thing-video/deploys

phazonoverload commented 4 years ago

Thank you so very much for this @RascalTwo! This is a great contribution, and it's good to see you free up the main thread with a worker. However, when testing, I wasn't able to get the output from the canvas into the publisher - instead it looked like this:

Screenshot showing green vertical bars instead of the camera output

Any idea why this might be the case?

phazonoverload commented 4 years ago

I'm using Chromium 86 in Brave 1.15 😄

RascalTwo commented 4 years ago

After testing, I'm quite confident it was due to the fact that I had disabled alpha in the canvas contexts, so hopefully this has resolved it for you also.

gif showing it working in Brave

Only in hindsight I realized that since the fake device is all green, the greenscreen effect isn't actually visible above - so the below gif actually displays the effect working.

git showing it working in Brave, but with the effect actually visible

Additionally, I've brought it up to date with the latest main commit - 685da9b - and added some basic styles that should match with the rest of the page.

Unrelated, but turns out my errors in chromium browsers with opentok were due to the low quality of my USB webcam, hence I used a fake device/video to demonstrate it in Brave.

phazonoverload commented 3 years ago

Hey! I tried this but still couldn't get it to work. This time, I can't see anything except the overlay image, and my chroma min and max are sensible. This is still an excellent piece of work, and I'll mark it as valid for the sake of Hacktoberfest.

phazonoverload commented 3 years ago

If you want to jump on a call to try and debug this let me know. I'll close this for now unless you want to do any further work 💖

RascalTwo commented 3 years ago

I'm glad the initial error was truly solved, wasn't able to be there live but I did watch your VOD.

The issue you had with pulling my PR was likely due to the fact that I rebased my changes to match the updated styles at the time, so this time I didn't do that so you shouldn't encounter the same issue again.

As for the chroma-keying not working, it appears that's due to the rigidity/inaccuracy of the algorithm I choose, so I've swapped it out for a different one that I can get working on just about any greenscreen video - even yours, even at the low resolution I was able to scrape from the VOD.

After seeing how much performance really drops in the VOD, I also decided to make performance a priority, so along with various micro optimizations, it's now much faster - probably as fast as I personally can make it without getting WebAssembly/WebGL involved.

Just to be safe though, in case performance is still lackluster, I've added options to allow for customization of the output - turning down the output resolution allows for higher frame rates.

Now I'm no Vue expert, so while I've got it all working, It's likely I've broken some Vue rules or missed some things that Vue offers to make certain things much easier, but hopefully not too many though.

I personally follow these steps to get each background working:

I was only able capture your webcam from the VOD at 320x180, hence the low quality, but as you see with the second example - with a resolution of 1920x1080 - it does work when the quality is higher.

working on Kevin webcam and 1080p real life sample

Lost some quality to get the .gif under 10 MB for the upload limit, here's the higher quality video if you're interested.

UI Clarification

I've also attempted to make the UI clearer, so hopefully the options are more obvious, but just in case here's a quick rundown:

Background

The Pause/Play button pauses or plays the input video - useful to fine tune the settings without having to deal with new frames or that one pixel you're trying to pick jumping around.

Output

With all the inputs being instantly reactive thanks to Vue, changes are now visible instantly instead of having to wait for focus to leave the input (the change event), so it's now easier and quicker to fine tune all these settings.

I appreciate the offer to jump into a call, and the time you've taken for this single issue, perhaps if these latest changes don't work as expected I'll take you up on that offer.