MV3 user's mic and cam permissions using iframe

esphoenixc commented 1 year ago

I am building a chrome extension that records user's cam, mic, and screen.

For the screen, I think I took care of it by using desktopCapture API in background.js.

The problem comes in for user's cam and mic.

I have an iframe injected into the shadow dom. From that iframe, I request for the permissions of the use of the user's cam and mic by using "navigator.mediaDevices.getUserMedia({audio: true, video:true})". The reason for requesting mic and cam permissions inside the iframe is because I want to ask for the permissions on behalf of my extension app (eg -> "[myExtension] wants to Use" instead of "www.google.com wants to use your camera and microphone") and hold onto the permissions across all tabs, only asking once for the permissions.

I want to display user's cam and mic streams in content script, not inside the iframe. In order for me to stream the user's cam and mic in content script, somehow I need to find a way to get the streams of the cam and mic from the iframe and send it over to content script and display the streams. I believe this is where chrome.tabCapture API comes in.

However, in manifest version 3, I cannot use chrome.tabCapture in backgrund.js (now also known as service workser). I think that's causing a major issue in this case.

As I researched more into this specific case, I've found that there's a new feature called, offscreen API.

Can I use offscreen API to achieve what I'm trying to achieve?

requesting permission from the permission.html inside the iframe, which will be created using offscreen API
use that permission in content script to use navigator.mediaDevices.getUserMedia() to get the streams of the user's cam and mic.

Is this even possible in manifest version 3 currently?

if what I wrote or am trying to accomplish is unclear, please let me know.

Here are the references I've gone over to solve this issue : https://bugs.chromium.org/p/chromium/issues/detail?id=1214847 https://bugs.chromium.org/p/chromium/issues/detail?id=1339382 https://stackoverflow.com/questions/74773408/chrome-tabcapture-unavailable-in-mv3s-new-offscreen-api https://stackoverflow.com/questions/66217882/properly-using-chrome-tabcapture-in-a-manifest-v3-extension https://github.com/GoogleChrome/chrome-extensions-samples/issues/627 https://github.com/w3c/webextensions/issues/170 https://groups.google.com/a/chromium.org/g/chromium-extensions/c/ffI0iNd79oo/m/Dnoj6ZIoBQAJ?utm_medium=email&utm_source=footer

patrickkettner commented 1 year ago

Hi @esphoenixc! I believe offscreenDocument should allow you to accomplish this? Let us know if you hit any issues with it.

I am tagging this as a potential sample the team could put together

M-SAI-SOORYA commented 1 year ago

Yes, it is possible to use the Offscreen API to achieve what you are trying to do in Manifest version 3.

To start, you can use the Offscreen API to create a hidden canvas element in your content script. You can then use navigator.mediaDevices.getUserMedia() to get the user's cam and mic streams inside your iframe and pass them to your background script.

From your background script, you can then use the chrome.runtime.sendMessage() method to send the streams to your content script, where you can display them on the hidden canvas element using CanvasRenderingContext2D.drawImage()

KiranNadig62 commented 1 year ago

What's the status of the sample? We are unable to get this working. Offscreen APIs cannot obtain media permissions from our testing.

jpmedley commented 1 year ago

I'm a little rusty on the issue, but I think you might be looking for this: https://developer.chrome.com/docs/extensions/mv3/screen_capture/

If so, I apologize for forgetting that we had an open ticket on this.

aakash232 commented 1 year ago

I'm a little rusty on the issue, but I think you might be looking for this: https://developer.chrome.com/docs/extensions/mv3/screen_capture/

If so, I apologize for forgetting that we had an open ticket on this.

@jpmedley The above solution will be wrt some specific tab. (Getting stream from tab) Main goal is as stated in the issue :-

The reason for requesting mic and cam permissions inside the iframe is because I want to ask for the permissions on behalf of my extension app (eg -> "[myExtension] wants to Use" instead of "www.google.com wants to use your camera and microphone") and hold onto the permissions across all tabs, only asking once for the permissions.

How do we get access to Media from the user's microphone using the offscreen API? I am talking about something like this.

(Inside offscreen.js)

function getUserPermission() {
  return new Promise((resolve, reject) => {
    console.log("Getting user permission for microphone access...");

    navigator.mediaDevices
      .getUserMedia({ audio: true })
      .then((response) => {
        if (response.id) 
            resolve();
      })
      .catch((error) => {
        console.error("Error requesting microphone permission:", error);
        if (error.message === "Permission denied") {
          reject("MICROPHONE_PERMISSION_DENIED");
        }
        reject(error);
      });
  });
}

Currently, this throws a NotAllowedError: Failed due to shutdown

aakash232 commented 1 year ago

@esphoenixc

Found this alternative for your requirement. (Without offscreen API)

Attaching steps for reference if anyone needs a possible workaround until offscreen api stuff is clear.

This will ask for the permissions on behalf of extension app (eg -> "[myExtension] wants to Use" instead of "www.google.com wants to use your microphone") and hold onto the permissions across all tabs, only asking once for the permissions.

In your Content Script (say, `injectDOM.js`)

Create and inject an iframe to the page where extension will be used.

const iframe = document.createElement("iframe");
iframe.setAttribute("hidden", "hidden");
iframe.setAttribute("id", "permissionsIFrame");
iframe.setAttribute("allow", "microphone");
iframe.src = chrome.runtime.getURL("requestPermissions.html");
document.body.appendChild(iframe);

This will have the src file requestPermissions.html. The script file for this page, requestPermissions.js will have the code to request permission.

requestPermissions.html

<!DOCTYPE html>
<html>
  <head>
    <title>Request Permissions</title>
    <script>
      "requestPermissions.js";
    </script>
  </head>
  <body>
    <!-- Display loading or informative message here -->
  </body>
</html>

requestPermissions.js

/**
 * Requests user permission for microphone access.
 * @returns {Promise<void>} A Promise that resolves when permission is granted or rejects with an error.
 */
async function getUserPermission() {
  return new Promise((resolve, reject) => {
    console.log("Getting user permission for microphone access...");

    // Using navigator.mediaDevices.getUserMedia to request microphone access
    navigator.mediaDevices
      .getUserMedia({ audio: true })
      .then((stream) => {
        // Permission granted, handle the stream if needed
        console.log("Microphone access granted");
        resolve();
      })
      .catch((error) => {
        console.error("Error requesting microphone permission", error);

        // Handling different error scenarios
        if (error.name === "Permission denied") {
          reject("MICROPHONE_PERMISSION_DENIED");
        } else {
          reject(error);
        }
      });
  });
}

// Call the function to request microphone permission
getUserPermission();

In `manifest.json`

"web_accessible_resources": [
    {
      "resources": ["requestPermissions.html", "requestPermissions.js"],
      "matches": ["<all_urls>"]
    }
],

juxnpxblo commented 1 year ago

I figured that you're able to get microphone stream on offscreen if user had previously consented to microphone permission for your extension, however, an offscreen document can't ask permission itself, so you must get it from some other context

on my demo using an action popup, I setup a popup.html with a script tag sourcing from popup.js, and in popup.js I had a getUserMedia({ audio: true }) call, which triggered the permission prompt " wants to use your microphone" on first time

after I consented, I was able to call getUserMedia({ audio: true }) from my offscreen document and successfully get a microphone stream; before consenting, I would get a DOMException: Failed due to shutdown

aakash232 commented 1 year ago

I figured that you're able to get microphone stream on offscreen if user had previously consented to microphone permission for your extension, however, an offscreen document can't ask permission itself, so you must get it from some other context

on my demo using an action popup, I setup a popup.html with a script tag sourcing from popup.js, and in popup.js I had a getUserMedia({ audio: true }) call, which triggered the permission prompt " wants to use your microphone" on first time

after I consented, I was able to call getUserMedia({ audio: true }) from my offscreen document and successfully get a microphone stream; before consenting, I would get a DOMException: Failed due to shutdown

True. Post consent, we can use getUserMedia({ audio: true }) in offscreen API. That's what I am using currectly.

Only thing was, your way of asking for consent (via popup.js) triggers "www.google.com/*** wants to use your microphone" instead of "[myExtension] wants to Use". Correct me if I am wrong.

Thanks

athioune commented 1 year ago

Hi @aakash232,

Do you happen to have a full example of your solution ? (Because with MV3 I always have a not allowed error)

Thank you !

aakash232 commented 1 year ago

Hi @aakash232,

Do you happen to have a full example of your solution ? (Because with MV3 I always have a not allowed error)

Thank you !

@athioune I will give an example scenario from my side. Let me know if you needed this or something else.

OBJECTIVE

GOAL : Get mic permissions wrt extension's context and record audio via offscreen API

ISSUE : Getting not allowed error if we request mic permissions inside offscreen doc

SOLUTION : Get permissions via external iframe, once consent is received handle audio capture and other tasks inside offscreen doc.

STEPS

Step 1 : To get permissions, follow my previous reply steps

Step 2 : Handle recording. (All the below steps can be achieved via offscreen doc)

Create offscreen doc initially

  await chrome.offscreen
    .createDocument({
      url: offscreen.html,
      reasons: [USER_MEDIA],
      justification: "keep service worker running and record audio",
    })

Inside offscreen.js

Check status of mic permissions

      if (result.state === "granted") {
        // we have the permissions, continue next steps
        handleRecording();
      } else if (result.state === "prompt") {
        // prompt user to get the permissions
        // Call the function in requestPermissions.js to request microphone permission [ getUserPermission() ]       
      } else if (result.state === "denied") {
        // permissions denied by user
      }
    });

In handleRecording() you can get audio input devices.

navigator.mediaDevices
      .enumerateDevices()
      .then((devices) => {
        // Filter the devices to include only audio input devices
        const audioInputDevices = devices.filter(
          (device) => device.kind === "audioinput"
        );
       // Use them accordingly
      })

Once we have the input devices, then we can capture audio from the device ID you wish to

const deviceId = audioInputDevices[0].deviceId;

    navigator.mediaDevices
      .getUserMedia({
        audio: {
          deviceId: { exact: deviceId },
        },
      })
      .then((audioStream) => {
           //get audio stream from your selected input device

         //pass the same for new media recorder instance 
          mediaRecorder = new MediaRecorder(audioStream);

          mediaRecorder.ondataavailable = (event) => {        
              chunks = [event.data];
             //handle chunks as needed            
          };

          mediaRecorder.onstop = handleStopRecordingFunction;

          //set time slice to capture data every X milliseconds
          mediaRecorder.start(5000);     
      });

athioune commented 1 year ago

Thank you very much @aakash232 it worked !

ddematheu commented 6 months ago

Sorry to revive this, but when I try to trigger permissions flow from the popup, I see an error: "Permissions Dismissed". The pop up is not even coming up. Anyone seen this?

aakash232 commented 6 months ago

Sorry to revive this, but when I try to trigger permissions flow from the popup, I see an error: "Permissions Dismissed". The pop up is not even coming up. Anyone seen this?

Received this error along with DOMException: Failed due to shutdown somewhere when I was trying to trigger the flow from the offscreen doc. To bypass this error, I went through the additional efforts of injecting iframe.

Few follow-up(s),

Did you check if the permissions aren't explicitly blocked by chrome for the extensions? (in manage extensions page).
The offscreen sample PR : Did the flow here helped you out?
If nothing works, can you please elaborate the flow via which you are showing the popup? (Is there something you are trying differently apart from the sample PR. Maybe more context can help in debugging).

rahulbansal16 commented 3 months ago

This works everywhere except the web WhatsApp. I get the error.

Permissions policy violation: microphone is not allowed in this document.

@esphoenixc

lammerfalcon commented 3 months ago

can someone share code how to implement basic permission access to mic? i am new to chrome extensions

PLtier commented 3 months ago

Hi, just wanted to share how I solved my problem: (answering two previous questions)

open extension page (can be from service worker after install )
prompt the user for permissions there
now it's possible to enable/disable camera in the other contexts (i.e. service worker/popup) This is imo least tricky way to do it - as others mentioned it's not possible to do it via other means. (except for iframe).

You can see example at https://chromewebstore.google.com/detail/straightenup-ai-ai-postur/nfhoegpkonllcaghgmhdmcpmebmocokf (i'm the author, but I think it's demonstrated well here)

GoogleChrome / chrome-extensions-samples