chrome tabCapture example

[ddenis1994]

hi am tiring to use the tabCapture API without success for a few days now

what I tried running the tab capture in the content - results in undefined to the API running the tab capture inside an iframe that the src is HTML that accessible to the extension (iFrame.src = chrome.runtime.getURL("gif.html");) -result in an error when the user clicks on a button to initiate the recorded Error starting tab capture

maybe someone can share code example for using the tab capture API in manifest version 3

thanks

[guest271314]

You should be able to open an <iframe> with URL set to chrome-extension://<id>/file.html by specifying the file in "web_accessible_resources" then use navigator.mediaDevices.getUserMedia() and navigator.mediaDevices.getDisplayMedia().

[guest271314]

The MV2 version does capture video, however suffers from MediaStreamTrack of kind video muting and unmuting due to unspecified Chrome implementation https://bugs.chromium.org/p/chromium/issues/detail?id=931033, https://bugs.chromium.org/p/chromium/issues/detail?id=1099280, https://bugs.chromium.org/p/chromium/issues/detail?id=1100746, https://github.com/w3c/mediacapture-screen-share/issues/141 resulting in the video not playing at HTML <video> element, rather is a freeze frame of the last time the user focused on the tab being shared.

What are you trying to achieve?

Can you post the code you are testing here?

[ddenis1994]

I am trying to capture the tab video for later edit share and upload in the end, I solved it by creating streamID inside the popup page and pass it to the content script for capturing .

[kwertop]

HI @ddenis1994, @guest271314 Could you please post any sample code for fetching mediaStream in content script. I've managed to get streamId from popup.html but when I'm doing:

navigator.mediaDevices.getUserMedia({ audio: { mandatory: { chromeMediaSource: 'desktop', //also tried by setting this to 'tab' chromeMediaSourceId: streamId } } }, (tabStream) => { // })

in the content script, I'm getting DOMException: Requested device not found error for chromeMediaSource = desktop while DOMException: Error starting tab capture error for chromeMediaSource = tab .

[guest271314]

@rahul0106 What specific media content are you trying to capture?

[kwertop]

@guest271314 I'm trying to capture audio in any live web based meeting conference app playing in a chrome tab. I'm able to capture user's audio input but I also need to capture the system audio.

[guest271314]

You can use getDisplayMedia() select Tab capture at UI to capture audio output of a tab. Chrome does not provide a means to capture entire system audio output, only audio output by a Chrome tab. On *nix you can utilize parec https://github.com/guest271314/captureSystemAudio/tree/master/native_messaging/capture_system_audio.

[guest271314]

@rahul0106 See also https://github.com/Lightcord/Lightcord/issues/31.

[kwertop]

@guest271314 By system audio, I meant the output audio of the tab. Like a speaker in a zoom meeting.

[kwertop]

I tried using getDisplayMedia(). It throws the following error: Failed to execute 'getDisplayMedia' on 'MediaDevices': Access to the feature "display-capture" is disallowed by permission policy desktopCapture permission is present in the manifest.json file.

[guest271314]

I meant the output audio of the tab.

Is audio playing in the tab at an <audio> element, or using AudioContext?

Can you post the code you are trying here?

[kwertop]

There isn't any <audio> element in the page source of a zoom call (not sure about AudioContext). This is what I'm trying:

navigator.mediaDevices.getUserMedia({
  audio: {
    mandatory: {
      chromeMediaSource: 'tab',
      chromeMediaSourceId: streamId // this is passed from popup.html using chrome.runtime.sendmessage
    }
  }
}, (tabStream) => {
    // do something with tabStream
});

[guest271314]

There isn't any <audio> element in the page source of a zoom call (not sure about AudioContext). This is what I'm trying:

navigator.mediaDevices.getUserMedia({
  audio: {
    mandatory: {
      chromeMediaSource: 'tab',
      chromeMediaSourceId: streamId // this is passed from popup.html using chrome.runtime.sendmessage
    }
  }
}, (tabStream) => {
    // do something with tabStream
});

Chrome only captures audio when the audio is playing in the tab, that is, <audio> and AudioContext audio destination node. For example, Chromium does not capture window.speechSynthesis.speak() when Google voices are not used, where speech dispatcher technically plays audio at system outside of the browser https://bugs.chromium.org/p/chromium/issues/detail?id=1185527, why this answer https://stackoverflow.com/questions/45003548/how-to-capture-generated-audio-from-window-speechsynthesis-speak-call/70665493#70665493 does not produce expected result.

The signature used in the code navigator.getUserMedia(constraints, successCallback, errorCallback); is deprecated https://developer.mozilla.org/en-US/docs/Web/API/Navigator/getUserMedia, see https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia.

As indicated in the comments linked

@rahul0106 See also Lightcord/Lightcord#31.

specifically note the constraints https://github.com/Lightcord/Lightcord/issues/31#issuecomment-922553830.

And the flags used in https://bugs.chromium.org/p/chromium/issues/detail?id=1195881#c19 for headless capture, e.g.,

--auto-select-desktop-capture-source="Entire screen"

manifest.json

{
  "name": "Tab audio capture",
  "version": "1.0",
  "manifest_version": 3,
  "permissions": [ 
    "tabs", 
    "activeTab", 
    "desktopCapture", 
    "scripting" 
  ],
  "background": {
    "service_worker": "background.js"
  },
  "action": {}
}

background.js

async function captureStream(streamId) {
  const stream = await navigator.mediaDevices.getUserMedia({
    video: {
      mandatory: {
        chromeMediaSource: 'screen',
        chromeMediaSourceId: streamId,
      },
    },
    audio: {
      mandatory: {
        chromeMediaSource: 'desktop',
        chromeMediaSourceId: streamId,
      },
    },
  });
  stream.removeTrack(stream.getVideoTracks()[0]);
  console.log(stream, stream.getTracks()[0]);
  const recorder = new MediaRecorder(stream);
  recorder.start();
  return new Promise((resolve) => {
    setTimeout(() => recorder.stop(), 1000 * 10);
    recorder.ondataavailable = (e) => {
      const blobURL = URL.createObjectURL(e.data);
      resolve(blobURL);
      console.log(blobURL);
    };
  });
}

chrome.action.onClicked.addListener(async (tab) => {
  console.log(tab);
  const { streamId, options } = await new Promise((resolve) => {
    chrome.desktopCapture.chooseDesktopMedia(
      ['tab', 'audio'],
      tab,
      async (streamId, options) => {
        resolve({ streamId, options });
      }
    );
  }).catch((err) => console.error(err));
  console.log(streamId, options);
  const [{frameId, result}] = await chrome.scripting.executeScript({
    target: {
      tabId: tab.id,
    },
    world: 'MAIN',
    args: [streamId],
    func: captureStream,
  });
  console.log(frameId, result);
});

audio-capture.zip

[LiuShuaiyi]

Is there anyway to start capturing the current tab directly (video only)? If I understand correctly navigator.mediaDevices.getUserMedia always pops up a window for users to select what to capture. What if I don't want that window? What if I just want to start capture the current active tab right away?

[guest271314]

They have designed the API to limit that capability. Should be possible using a local application.

[guest271314]

@LiuShuaiyi You can try with this flag https://peter.sh/experiments/chromium-command-line-switches/#auto-select-desktop-capture-source --auto-select-desktop-capture-source="Tab".

[LiuShuaiyi]

Okay I finally got a working prototype with the same approach as said in OP's comment: https://github.com/GoogleChrome/chrome-extensions-samples/issues/627#issuecomment-899104997.

• Use chrome.tabCapture.getMediaStreamId to get stream id in popup.
• Pass the stream id to content script with chrome.tabs.sendMessage.
• In content script's message handler, use navigator.mediaDevices.getUserMedia to start capture.

With this approach, capture starts immediately on the active tab without the media select popup window.

[donething]

Okay I finally got a working prototype with the same approach as said in OP's comment: #627 (comment).

• Use chrome.tabCapture.getMediaStreamId to get stream id in popup.
• Pass the stream id to content script with chrome.tabs.sendMessage.
• In content script's message handler, use navigator.mediaDevices.getUserMedia to start capture.

With this approach, capture starts immediately on the active tab without the media select popup window.

In the third step, Why I can not use navigator.mediaDevices.getUserMedia to start capture in content script's message handler.

The Chrome hint "DOMException: Error starting tab capture".

[guest271314]

Can you post your code here?

[donething]

Can you post your code here?

let streamer;
let audioCtx;
let source;
let gainNode;

chrome.runtime.onConnect.addListener(port => {
  port.onMessage.addListener(async msg => {
    audioCtx = new AudioContext();
    try {
      // The Chrome hint "DOMException: Error starting tab capture".
      streamer = await navigator.mediaDevices.getUserMedia({
        audio: {
          mandatory: {
            chromeMediaSource: 'tab',
            // this is passed from popup.html using chrome.runtime.sendmessage
            chromeMediaSourceId: msg.streamID
          }
        }
      });
    } catch (e) {
      console.log(e);
    }

    // bind audiocontext...
  });
});

[guest271314]

@donething The code in https://github.com/GoogleChrome/chrome-extensions-samples/issues/627#issuecomment-1013604912 works as expected.

[guest271314]

@donething How does the user activate the popup? Can you post you complete code here?

[donething]

@donething How does the user activate the popup? Can you post you complete code here?

I'm sorry, I had deleted those code, I cannot post them now.

I had test https://github.com/GoogleChrome/chrome-extensions-samples/issues/627#issuecomment-1013604912 on my brower, It works.

Thank you.

[zhw2590582]

@LiuShuaiyi @ddenis1994 After I use the method you mentioned, I can successfully get the audio, but the page will become silent, I want it to continue to play the sound, do you have a way to do this

[zhw2590582]

@guest271314 @ddenis1994 @donething

test-the-audio.zip You can download and test it, I don't know where I'm doing wrong, can you help me take a look, thanks a lot:

manifest.json

{
  "name": "Test the audio",
  "version": "1.0.0",
  "description": "Sound can be recorded, but the tab is muted",
  "manifest_version": 3,
  "permissions": ["tabs", "activeTab", "scripting", "tabCapture"],
  "action": {
    "default_popup": "popup.html"
  }
}

popup.js

// Get the current id
chrome.tabs.query(
  {
    active: true,
    currentWindow: true,
  },
  (tabs) => {
    const tabId = tabs[0].id;
    // Get the streamId
    chrome.tabCapture.getMediaStreamId(
      {
        consumerTabId: tabId,
      },
      (streamId) => {
        // Load the content.js
        chrome.scripting.executeScript(
          {
            target: { tabId },
            files: ["content.js"],
          },
          () => {
            // Send the streamId to the tab
            chrome.tabs.sendMessage(tabId, streamId);
          }
        );
      }
    );
  }
);

chrome.runtime.onMessage.addListener(async (streamId) => {
  console.log("streamId--->", streamId);

  // Get the stream
  const stream = await navigator.mediaDevices.getUserMedia({
    audio: {
      mandatory: {
        chromeMediaSource: "tab",
        chromeMediaSourceId: streamId,
        echoCancellation: true,
      },
    },
  });

  console.log("stream--->", stream);

  // Now I successfully recorded the sound, but the page is also muted
  // -------------------------------------------------------------------
  // Connected to an audio player, but still no sound, why ?????????????
  // -------------------------------------------------------------------

  const audioContext = new AudioContext();
  const mediaStream = audioContext.createMediaStreamSource(stream);
  mediaStream.connect(audioContext.destination);
});

[guest271314]

@guest271314 @ddenis1994 @donething

test-the-audio.zip You can download and test it, I don't know where I'm doing wrong, can you help me take a look, thanks a lot:

manifest.json

{
  "name": "Test the audio",
  "version": "1.0.0",
  "description": "Sound can be recorded, but the tab is muted",
  "manifest_version": 3,
  "permissions": ["tabs", "activeTab", "scripting", "tabCapture"],
  "action": {
    "default_popup": "popup.html"
  }
}

popup.js

// Get the current id
chrome.tabs.query(
  {
    active: true,
    currentWindow: true,
  },
  (tabs) => {
    const tabId = tabs[0].id;
    // Get the streamId
    chrome.tabCapture.getMediaStreamId(
      {
        consumerTabId: tabId,
      },
      (streamId) => {
        // Load the content.js
        chrome.scripting.executeScript(
          {
            target: { tabId },
            files: ["content.js"],
          },
          () => {
            // Send the streamId to the tab
            chrome.tabs.sendMessage(tabId, streamId);
          }
        );
      }
    );
  }
);

chrome.runtime.onMessage.addListener(async (streamId) => {
  console.log("streamId--->", streamId);

  // Get the stream
  const stream = await navigator.mediaDevices.getUserMedia({
    audio: {
      mandatory: {
        chromeMediaSource: "tab",
        chromeMediaSourceId: streamId,
        echoCancellation: true,
      },
    },
  });

  console.log("stream--->", stream);

  // Now I successfully recorded the sound, but the page is also muted
  // -------------------------------------------------------------------
  // Connected to an audio player, but still no sound, why ?????????????
  // -------------------------------------------------------------------

  const audioContext = new AudioContext();
  const mediaStream = audioContext.createMediaStreamSource(stream);
  mediaStream.connect(audioContext.destination);
});

I tested on Chromium 102.

I reproduced the tab muting when when getUserMedia() is executed.

Looks like a bug to me https://crbug.com.

The tab audio should not be muted when getUserMedia() is called.

Recording the audio does capture sound.

I re-tested the code at https://github.com/GoogleChrome/chrome-extensions-samples/issues/627#issuecomment-1013604912. The tab is not muted, the recording is garbled.

This error is reported at chrome://extensions:

Uncaught (in promise) Error: The message port closed before a response was received.

[zhw2590582]

@guest271314

The following error can be ignored

Uncaught (in promise) Error: The message port closed before a response was received.

test-the-audio-v2.zip

I use the code you said, the audio can be successfully recorded, and the page can play the sound normally

But I don't want the browser to evoke the tab selection popup again, I want to record directly on the current page

{
  "name": "Test the audio",
  "version": "1.0.0",
  "description": "Sound can be recorded, but the tab is muted",
  "manifest_version": 3,
  "permissions": ["tabs", "activeTab", "scripting", "desktopCapture"],
  "action": {
    "default_popup": "popup.html"
  }
}

popup.js

// Get the current id
chrome.tabs.query(
  {
    active: true,
    currentWindow: true,
  },
  (tabs) => {
    const tab = tabs[0];
    const tabId = tab.id;

    // Get the streamId from desktopCapture
    chrome.desktopCapture.chooseDesktopMedia(
      ["tab", "audio"],
      tab,
      (streamId) => {
        // Load the content.js
        chrome.scripting.executeScript(
          {
            target: { tabId },
            files: ["content.js"],
          },
          () => {
            // Send the streamId to the tab
            chrome.tabs.sendMessage(tabId, streamId);
          }
        );
      }
    );
  }
);

content.js

const audioContext = new AudioContext();

chrome.runtime.onMessage.addListener(async (streamId) => {
  console.log("streamId--->", streamId);

  // Get the stream
  const stream = await navigator.mediaDevices.getUserMedia({
    video: {
      mandatory: {
        chromeMediaSource: "screen",
        chromeMediaSourceId: streamId,
      },
    },
    audio: {
      mandatory: {
        chromeMediaSource: "desktop",
        chromeMediaSourceId: streamId,
      },
    },
  });

  stream.removeTrack(stream.getVideoTracks()[0]);

  console.log("stream--->", stream);

  // Here you can successfully record the sound, and the page will not mute
  // -------------------------------------------
  // If I don't create the recorder node, the recording is garbled, it's weird
  // -------------------------------------------

  const mediaStream = audioContext.createMediaStreamSource(stream);

  const recorder = audioContext.createScriptProcessor(0, 1, 1);
  recorder.onaudioprocess = (event) => {
    const inputData = event.inputBuffer.getChannelData(0);
    console.log("audio--->", inputData);
  };

  mediaStream.connect(recorder);
  recorder.connect(audioContext.destination);
});

[guest271314]

But I don't want the browser to evoke the tab selection popup again, I want to record directly on the current page

What do you mean by

again

?

You can try the following flag on your own machine.

@LiuShuaiyi You can try with this flag https://peter.sh/experiments/chromium-command-line-switches/#auto-select-desktop-capture-source --auto-select-desktop-capture-source="Tab".

There are other similar flags.

I don't think you can bypass the capture UI using extension code alone.

This code from your first example

  // Get the stream
  const stream = await navigator.mediaDevices.getUserMedia({
    audio: {
      mandatory: {
        chromeMediaSource: "tab",
        chromeMediaSourceId: streamId,
        echoCancellation: true,
      },
    },
  });

that mutes the audio output at the tab is a bug I suggest you file an issue about at https://crbug.com.

[zhw2590582]

@guest271314 ok, thanks a lot for your help

[dima11221122]

Is it possible to capture tab including navigation. Aformentioned approach can be used only if a webpage doesn't refresh.

[guest271314]

You can capture the entire screen.

[mirkonasato]

background.js

async function captureStream(streamId) {
  const stream = await navigator.mediaDevices.getUserMedia({ /*...*/ });

@guest271314 did you manage to actually run that code in background.js? navigator.mediaDevices is not available in a service worker as far as I can tell.

[guest271314]

@mirkonasato navigator.mediaDevices is run in the foreground.

What are you trying to achieve?

[mirkonasato]

@mirkonasato navigator.mediaDevices is run in the foreground.

You're right, sorry. I hadn't noticed that captureStream is passed to executeScript.

What are you trying to achieve?

Basically trying not to run the MediaRecorder in the same tab being recorded, otherwise if the user clicks a link loading a different page the recording is lost. (As @dima11221122 also mentioned.)

According to the tabCapture docs

Capture is maintained across page navigations within the tab

but that clearly doesn't work if the script doing the recording is injected into the same tab page.

[guest271314]

Basically trying not to run the MediaRecorder in the same tab being recorded, otherwise if the user clicks a link loading a different page the recording is lost. (As @dima11221122 also mentioned.)

I don't understand. Are you talking about navigating away from a page while still recording the ancestor page?

[guest271314]

Basically trying not to run the MediaRecorder in the same tab being recorded

It is possible to record a tab from a different tab. Can you kindly clarify what you are trying to achieve?

[mirkonasato]

I don't understand. Are you talking about navigating away from a page while still recording the ancestor page?

Exactly.

It is possible to record a tab from a different tab.

Yes that's an option, thanks. Although opening a new tab to do the recording seems a bit weird from a user experience point of view.

Can you kindly clarify what you are trying to achieve?

I'm experimenting to see what's possible really. The goal is simply to record a video of the user interacting with a website, a bit like a screencast. But I'm trying to figure out how this could work in terms of user interface.

tabCapture only works in the popup apparently, so the user needs to open the extension popup first. Then I was thinking of having a Record button in the popup, and when the user clicks it inject the recording code into the tab page. But as mentioned that doesn't work if the user navigates to a different page.

I'm starting to think it would be easier to build a regular web app rather than an extension. All the screen recording extensions I looked at use Manifest V2, not V3.

[guest271314]

It is already possible to use getDisplayMedia() to record any tab.

[guest271314]

I'm starting to think it would be easier to build a regular web app rather than an extension. All the screen recording extensions I looked at use Manifest V2, not V3.

The requirement is possible with an MV3 extension as well. The user does not have to remain focued on the tab for recording to continue while the user opens other tabs.

If you want to record a page in the background you can open an <iframe> or a new window to record.

[killergerbah]

Using @zhw2590582 's work as a reference (https://github.com/zhw2590582/chrome-audio-capture) I re-implemented audio capture when I upgraded to MV3 (https://github.com/killergerbah/asbplayer/blob/main/extension/src/services/BackgroundPageAudioRecorder.ts). The trick is in recognizing that you can access the tabCapture API from an inactive tab created from HTML packaged with the extension. The UX is still worse than MV2 since you need the extra tab but with some effort it's possible to keep that tab somewhat "invisible."

[Deewde]

Okay I finally got a working prototype with the same approach as said in OP's comment: #627 (comment).

• Use chrome.tabCapture.getMediaStreamId to get stream id in popup.
• Pass the stream id to content script with chrome.tabs.sendMessage.
• In content script's message handler, use navigator.mediaDevices.getUserMedia to start capture.

With this approach, capture starts immediately on the active tab without the media select popup window.

But then the original tab sound is muted while recording and there's no way to unmute it. The sound is present in the recording, though. This approach is unusable for any use case I can think of.

[guest271314]

But then the original tab sound is muted while recording and there's no way to unmute it.

What do you mean by "original tab"? What issue are you having? Kindly explain.

[Deewde]

Apologies if I wasn't clear enough - I meant "the original live sound of the tab being recorded" is being muted. I've posted a detailed description of the bug on the Chromium Extensions group.

[guest271314]

@Deewde

I've posted a detailed description of the bug on the Chromium Extensions group.

They, whoever they are, decided to ban me from contributing to that group, so I can't post or try to help you over there.

I have not experienced the captured Tab audio being muted. Can you create a gist or GitHub repository with the complete code you are testing?

To use chrome.tabCapture because it's less invasive than chrome.desktopCapture or navigator.mediaDevices.getDisplayMedia

That is pure opinion. Just use getDisplayMedia() and be done with the matter?

[Deewde]

@guest271314 Here you go, the more detailed description on StackOverflow. I only want to use getDisplayMedia/desktopCapture as a last resort because of the intrusive prompts. I've also tried to record the speaker output, but that failed too. I see you've also contributed to that exploration yourself, maybe you can provide more details on how you managed to get it working. Didn't work for me.

[guest271314]

Chrome and Cromium do not support capture of monitor devices on Linux. And 'audiooutput' does not mean system audio output, see also https://github.com/guest271314/SpeechSynthesisRecorder/issues/17. Neither does systemAudio actually mean system audio https://github.com/GoogleChrome/developer.chrome.com/issues/3957.

So in spite of specification and implementation authors of Chrome using such language as 'audiooutput' and 'systemAudio' they themselves know Chrome does not support capturing speakers or headphones https://bugs.chromium.org/p/chromium/issues/detail?id=1136480#c13.

I suggest remapping speakers to microphone, something like

pactl load-module module-remap-source \
  master=@DEFAULT_MONITOR@ \
  source_name=speakers source_properties=device.description=Speakers \
&& pactl set-default-source speakers

or PipeWire solution found here https://github.com/edisionnano/Screenshare-with-audio-on-Discord-with-Linux.

I use Native Messaging to capture output to speakers and headphones https://github.com/guest271314/captureSystemAudio/tree/master/native_messaging/capture_system_audio.

[Deewde]

Ok, so not working. I aim to upload this to the store, so any client-side changes are out of the question. How about the bug I posted here, what do you think about that? Can you reproduce it yourself? In essence passing a stream id from an iframe to a content script, then using getUserMedia to get that stream, resulting in a good recording but a muted tab.

[guest271314]

Kindly upload the entire extension to a gist or GitHub repository with steps to reproduce. (https://stackoverflow.com/help/minimal-reproducible-example). Then I can test.

[guest271314]

@Deewde To avoid prompts altogether you can

1. Set the default audio device that you intend to capture;
2. Launch Chrome with --use-fake-ui-for-media-stream flag

I am not sure what is causing the muting behaviour you described without a repoducible example.

[Deewde]

I will provide an example as soon as I'm able to. I plan on launching this on the store, so no client-side modifications, including launching with flags.

On Fri, Dec 30, 2022, 16:22 guest271314 @.***> wrote:

@Deewde https://github.com/Deewde To avoid prompts altogether you can

1. Set the default audio device that you intend to capture;
2. Launch Chrome with --use-fake-ui-for-media-stream flag

I am not sure what is causing the muting behaviour you described without a repoducible example.

— Reply to this email directly, view it on GitHub https://github.com/GoogleChrome/chrome-extensions-samples/issues/627#issuecomment-1367945144, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHKFQZQNHN3BHJMGJWERQBTWP3V2RANCNFSM5CAGIHZQ . You are receiving this because you were mentioned.Message ID: @.***>