Need help with video overlay?

[zzzsharazzz]

My question is: Can I use a video as a background, then put the Webcam preview on top of it?

I have a project which allows the user to stand in front of a green background, then the Webcam preview will show him/her with a video playing in the back (chromakey effect, like they do on weather forecast channels). I already read through all the examples and wonder if the above is possible or not. Anyone with experience can help me?

[sarxos]

@zzzsharazzz,

I wanted to answer "yes, you can", but I tried to implement this with vlcj by putting EmbeddedMediaPlayerComponent below WebcamPanel and making the webcam panel non-opaque (with some small changes in the code to support translucent painting). It would normally work fine, but no matter the effort, is was impossible :(

Everything was working almost as it should, but the webcam panel was not transparent when put on top of video panel.

I spent some time yesterday to play with the code and do some google searches and I found out that there are two kinds of graphical components in Java:

• Heavyweight - this is whole Java AWT including classes like Canvas, Panel, etc.
• Lightweight - this is whole Swing including classes like JPanel, JLabel, etc.

In general - the vlcj video rendering panel is created from heavyweight component and, from the other hand, the webcam panel from Webcam Capture API is a lightweight component. And the problem is the following - when heavy component is put under the light component, then the light component will not be transparent (source: link 1, link 2).

The bottom line is that:

• It's impossible with predefined components of vlcj, but
• It's probably possible with direct rendering which can be summarised by the following: first you get BufferedImage from video and then get another BufferedImage from webcam and then replace some specific color range with alpha and finally paint the second one on the first one. This will get you want you need.

If you find some other framework which render video on lightweight component then please let me know so I implement necessary changes into the Webcam Capture API to support translucent panel painting.

[zzzsharazzz]

Thanks a lot for your reply. Yes I also tried the vlcj method before but as you said, I couldn't make it work. I'm still looking for a way to do this, will post it here when I'm done. By the way, I really love your work, hope you could make it even better.

[zzzsharazzz]

Hello, just want to update, I finally did this using your suggestions, maybe it will help others like me.

I use JavaCV, this library has the class FFmpegFrameGrabber which is what I need to grab the frames and convert them to BufferedImage, then mix with the BufferedImage returned from Webcam and that's it.

The result is pretty smooth and I think it's better than translucent panel painting. Thanks a lot @sarxos

[sarxos]

Hi @zzzsharazzz,

Thank you for the feedback! So you finally used direct rendering approach. I had a hunch this will be the best approach in terms of this issue.

Just a side note for the ones who would like to do it with vlcj instead of JavaCV - the same is also possible with vlcj by using getSnapshot(int, int) method from EmbeddedMediaPlayer, but this specific method saves snapshot to a temporary file before it's returned as BufferedImage and therefore cause I/O to be a bottleneck in this solution (which will effectively cause video FPS degradation).