Use Chrome Desktop Capture as video source?

[UUoocl]

This looks awesome! I'm excited to try your project. I've been experimenting with mediapipe js, but have never used TouchDesigner.

I want to recommend the Chrome Desktop Capture as another option for bringing video into the mediapipe-touchdesigner.

In this repo I used Chrome's Desktop Capture as the video input source for landmark detection.
OBS-face-tracking-with-P5js

In this setup a webcam source is created in OBS, then a projector window is opened on that source. Chrome is used to capture the OBS projector window. Landmark data is sent from Chrome to OBS via websockets.

Using NDI, Spout/Syphon or OBS virtual camera accomplish the same purpose, but this options may be useful too.

Thanks for sharing this project!

[domisjustanumber]

Hey @UUooc, thanks so much for this suggestion! I've been trying to figure out an elegant way to get video from TouchDesigner to Chrome that doesn't require any external plugins and this might be that solution!

I'll have a tinker and see if I can get a similar chain running. In theory it should be possible with a Window COMP in TouchDesigner (make a new window with something in it), and then use that as the capture source... I'll see how the theory stacks up with reality

[UUoocl]

I've started a MediaPipe for OBS project. Electron js is used to run the MediaPipe models locally and send the landmarks to OBS. The landmarks are stored in a Text Source as JSON. The Advanced Scene Switcher plug-in reads the landmarks JSON to control OBS.

Thanks for the inspiration!

[domisjustanumber]

Oh wow super cool! Being able to trigger scene changes or effects in OBS with poses or hand gestures will be super fun - great work!

[domisjustanumber]

Just thought I'd drop an update in here after spending the day trying to get this to work with the embedded Chromium that TouchDesigner uses in case you've seen it work otherwise.

It seems that the command line options to allow auto-accept of window or screen sharing don't work in CEF, and as there's no mechanism to show menus with CEF, I can't get this working right now.

First up, it seems that chrome.desktopCapture only works if you're running a chrome extension (which I guess may be possible, but something to investigate later)

The more standard way of requesting a media source is navigator.mediaDevices.getDisplayMedia which works great when running the page in Chrome - you get a pop-up asking what you want to share, select the window you want, it all works.

Permission denied

When running in the CEF instance inside TouchDesigner, I get a permission denied error when trying to ask for a capture device. So far the options I've tried are:

• No options specified
• Enable Media Stream toggled on and off
• --auto-select-window-capture-source-by-title="MP_capture"
• --auto-select-desktop-capture-source="MP_capture"

(where MP_capture is the name of the window with some test content)

Limited success

If I add --use-fake-ui-for-media-stream I do get a video stream (yay!) but it's a whole screen of a random monitor, not the window name I've specified, and no apparent way to change it.

Other things to try

Some reports said adding the option --enable-chrome-runtime would run a full version of Chrome with all the navigation UI and that may allow me to try other things, but that doesn't seem to work in the TouchDesigner embedded version.

@UUoocl if you know of any other option flags I could try, or other ways to make this work that would be great. It sounds like someone else found the issue and submitted a PR that got rolled into the Chromium code last month, so maybe it'll get fixed in CEF: https://www.magpcss.org/ceforum/viewtopic.php?f=6&t=19676

[domisjustanumber]

Looping back to this - I just tried the same things in TouchDesigner 2023 which now has CEF 115 (vs CEF 100 in TouchDesigner 2022) and it does slightly different things with different toggles, but the end result is the same - I can only get complete, all-screens sharing working, I don't seem to have any ability to force only a specific window to be captured.

[domisjustanumber]

I went down a rabbit hole chasing this one to see if it's anything in CEF we can manipulate. The short version seems to be that

--auto-select-window-capture-source-by-title="MP_capture" --auto-select-desktop-capture-source="MP_capture"

Don't do anything in CEF - they only affect full Chrome browser.

I found this little ditty in the CEF source code that says if no ID is asked for, the default is to share Screen -1 (entire screen)

        media_id =
            content::DesktopMediaID(content::DesktopMediaID::TYPE_SCREEN,
                                    -1 /* webrtc::kFullDesktopScreenId */);
      }
      video_devices.emplace_back(request_.video_type, media_id.ToString(),
                                 "Screen");
    }

As a (probably sensible) security feature, JS code running in a browser is not allowed to ask for the sharing of any specific Window or Screen... so for now I'm not sure if there's a way to make this work.

One workaround I considered is do as above and crop the entire screen to only the window we want, but for that to work the window we want to capture from would need to be fully visible on the screen at all times, which is less than ideal. I am hoping there's another way that would work with the window hanging off the edge of a screen so it's only 1 or 2 pixels visible, but the window is drawn by the GPU and thus could be captured by Chrome (this works in full Chrome browser, so that part is possible).

I've raised this ticket with the CEF team to see what they say... https://github.com/chromiumembedded/cef/issues/3667

[simoher]

Hello there! Been working with all kind of Streaming since streaming online radio in 08.. Much happen since yes!.. insent the goto standart for realtime www streaming now if im not wrong? Know its supported in breave wich is based on Chrome.

Good day/night, thanks for working on this project, hope to share my progress and contribute more when i got more of the most expensive "thing".

Love'n light Simolator

---

From: Dom Scott @.> Sent: Tuesday, March 12, 2024 11:05:10 PM To: torinmb/mediapipe-touchdesigner @.> Cc: Subscribed @.***> Subject: Re: [torinmb/mediapipe-touchdesigner] Use Chrome Desktop Capture as video source? (Issue #64)

I went down a rabbit hole chasing this one to see if it's anything in CEF we can manipulate. The short version seems to be that

--auto-select-window-capture-source-by-title="MP_capture" --auto-select-desktop-capture-source="MP_capture"

Don't do anything in CEF - they only affect full Chrome browser.

I found this little ditty in the CEF source codehttps://bitbucket.org/chromiumembedded/cef/src/d4cf19db29b016e3d781517bc3bd93b118e81b39/libcef/browser/media_access_query.cc?at=master#lines-167 that says if no ID is asked for, the default is to share Screen -1 (entire screen)

    media_id =
        content::DesktopMediaID(content::DesktopMediaID::TYPE_SCREEN,
                                -1 /* webrtc::kFullDesktopScreenId */);
  }
  video_devices.emplace_back(request_.video_type, media_id.ToString(),
                             "Screen");
}

As a (probably sensible) security feature, JS code running in a browser is not allowed to ask for the sharing of any specific Window or Screen... so for now I'm not sure if there's a way to make this work.

One workaround I considered is do as above and crop the entire screen to only the window we want, but for that to work the window we want to capture from would need to be fully visible on the screen at all times, which is less than ideal. I am hoping there's another way that would work with the window hanging off the edge of a screen so it's only 1 or 2 pixels visible, but the window is drawn by the GPU and thus could be captured by Chrome (this works in full Chrome browser, so that part is possible).

I've raised this ticket with the CEF team to see what they say... chromiumembedded/cef#3667https://github.com/chromiumembedded/cef/issues/3667

— Reply to this email directly, view it on GitHubhttps://github.com/torinmb/mediapipe-touchdesigner/issues/64#issuecomment-1992661838, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AA4XFQ674U6EO75ASDV5C73YX53ZNAVCNFSM6AAAAABCO2XYBKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJSGY3DCOBTHA. You are receiving this because you are subscribed to this thread.Message ID: @.***>

[UUoocl]

In Electron the browser can not access the Get User Media directly either. The work around is to use node to query the available devices then pass the results to the browser. https://www.electronjs.org/docs/latest/api/desktop-capturer The window "ID" has been more reliable then using the window "name" when setting the video stream.

Another option to bring video in could be webRTC. webRTC can run on the localhost and only needs a websocket server to connect 2 browsers.

There is an example of connecting a browser to the OBS CEF browser MediaPipe for OBS project with webRTC. https://github.com/UUoocl/MediaPipe_for_OBS

[domisjustanumber]

Ohhh thank you for those links @UUoocl !

It looks like the secret sauce that Electron is using is a couple of Chrome-specific options you can pass to getUserMedia called chromeMediaSource and chromeMediaSourceId

When I get some time to tinker next, I can try them out and see if I can pull the list of Window IDs, then push the relevant ID to getUserMedia with the chromeMediaSourceId option... that might just do it.

[IcyFold]

Hi @domisjustanumber, did you have to enable the "enable-usermedia-screen-capturing" switch in cef for the desktop capture to work?

[robinma]

I use chrome extension, can get label video steam.

`const getPageStream = async () => {

const id = await chrome.tabCapture.getMediaStreamId(); console.log('id', id); let stream = await navigator.mediaDevices.getUserMedia({ video: { mandatory: { chromeMediaSource: 'tab', chromeMediaSourceId: id, }, }, audio: false, }); // const [videoTrack] = stream.getVideoTracks(); streamObj = stream; }`