search

PhilippMatthes / neural-greenscreen

Real time background replacement on a mac os driven webcam using the DeepLabV3 neural network for image segmentation and the native CoreMediaIO DAL framework of Mac OS.
https://philippmatthes.medium.com/how-to-create-a-system-wide-plugin-for-mac-os-which-removes-the-background-of-the-webcam-2cb88a94dc7b
MIT License
39 stars 6 forks source link

How to call plugin from C++ or swift application to feed the buffer #1

Closed Raj123456788 closed 4 years ago

Raj123456788 commented 4 years ago

trying to get an answer for this over a week now. Any guidance is appreciated.

PhilippMatthes commented 4 years ago

Please see the implementation of neural-greenscreen/Interface/Virtual Camera/Stream.swift (Especially Lines 230+) for the relevant code on how to pipe image data (from Swift or C++, as you wish) into the pixelbuffer. Please look at https://github.com/seanchas116/SimpleDALPlugin for another example. If you have any further questions which are actually related to this project, hit me up again.

Raj123456788 commented 4 years ago

Thanks. The plugin does not show up in quicktime player once added to the DAL folder.

PhilippMatthes commented 4 years ago

Try quitting the QuickTime Player with Cmd+Q and restarting it. Then it should show up.

Raj123456788 commented 4 years ago
Screen Shot 2020-09-23 at 10 21 05 AM

I have tried that still no luck. I have also looked into https://github.com/seanchas116/SimpleDALPlugin. There is no application that I can see how to send the buffer dataptr to the plugin. Running catlina 10.15.4

Raj123456788 commented 4 years ago

From the repo directory, run ./create-root-cert.sh and enter desired credentials for your signing certificate. Ran this command but it does not generate the rootCA.pem I can only see rootCA.key

Screen Shot 2020-09-23 at 10 31 32 AM
Raj123456788 commented 4 years ago

what changes I need to make the app in order to make it work with https://github.com/seanchas116/SimpleDALPlugin ? Atleast I can get started with that.

Raj123456788 commented 4 years ago

got rootCA.pem working

PhilippMatthes commented 4 years ago

This seems to be a trial and error issue. To clarify this in advance: This repository is a fork of SimpleDALPlugin and may overlap with the functionalities and addresses used to create the plugin, which is why you probably only see one plugin and not both. It extends the basic functionalities of the example DAL plugin by image segmentation based background replacement. This is done by periodically fetching data from a node js app (after transferring the current camera image). For that to work one potentially needs an ssl certificate to communicate with the node js application (encrypted). That's what the ssl key generation is for.

To be fair, it took me a while to dig into the CoreMediaIO framework as well, because it is very complex. But I cannot tell you exactly which steps you need to take to achieve your goal, these steps you need to take yourself. For that, read the documentation carefully, maybe reach out for help on third party platforms such as stackoverflow, or look at the example code to catch the idea.

To be clear: I unfortunately cannot give you a detailed tutorial on how CoreMediaIO DAL works, that's on your part.

Raj123456788 commented 4 years ago

you were right I removed simple DAL and your plugin appeared. But the background does not show up after yarn start since I have not trusted the cert as I cannot find it :) can you please help me locating it so that I get it working?

Raj123456788 commented 4 years ago

got it working thanks!

Raj123456788 commented 4 years ago

Can you send me some CoreMediaIO DAL documentation/links if possible?

Raj123456788 commented 4 years ago

@PhilippMatthes Hi Philipp, I am trying to eliminate the video camera and just show the image. Tried commenting the video capture but that did not help. Any suggestions?

PhilippMatthes commented 4 years ago

I locked this conversation since it is no longer related to the original issue. Please file another separate issue if there is anything more. Thanks!

PhilippMatthes commented 4 years ago

As a final hint, try to understand where the separate pixelbuffers are combined into one. Then start from there and select only one of the pixelbuffers.