tabCapture with MV3

[aeharding]

Will #134 mean that audio/video capture will be restored in an "event" page? (Example: tabCapture API.)

(Since audio/video capture is completely unsupported in service workers: https://developer.chrome.com/docs/extensions/mv3/migrating_to_service_workers/#audio_vidio)

If so - I don't understand the limitation of 5 minutes for a maximum lifetime of the event page.

Right now, with MV2, I capture Blobs via MediaRecorder in the background page. Chrome is smart about managing those Blobs with memory/disk, as I need to temporarily store them before they can be transmitted over the network to the server.

In MV3's event page proposal from #134, at least I'll be able to access the tabCapture API (supposedly). But because event pages are still ephemeral, does that mean I'll have to serialize and deserialize multiple megabytes of video capture as base64 strings and store them in chrome.storage every couple seconds as new blobs come in (since even pages have a 5 minute lifetime)?

Isn't that really CPU intensive versus just leaving a persistent background page open?

[bershanskiy]

If you need to capture a specific tab, you can do it from the content script (which persists for the entire lifetime of the page). You might have a harder time capturing the entire desktop or a window.

If you have more specific practical questions, I probably can answer them.

[aeharding]

Thanks, however I need to send the data to the server once captured, and CORS in content scripts is enforced in MV3, correct?

Which means I'd still need to serialize the data and transmit to the background page so I can send to my server.

[bershanskiy]

One way you could do this is to open a regular tab (chrome.tabs.create() or chrome.windows.create()) and capture video and transmit it via this regular tab. Since this is just a standalone tab, it is not affected by CORS restrictions. Once you are done with capturing, just close the tab. If user stumbles upon this tab and decides to close it, just describe the reason for needing this tab. Does this make sense?

[bershanskiy]

Again, this might feel like a workaround (since user might see an extra tab), but it will work for now.

[aeharding]

Yeah, it's just a really, really awful user experience.

I propose:

Persistent event pages

This is like #134 except persistent. There should be limitations to prevent extension developers abusing this, or using it unnecessarily, such as:

1. The persistent event page can only be enabled via a direct browser extension interaction (like activeTab has). This would prevent extensions from having persistent background event pages unless the user directly interacts with the extension.
2. The browser may forcefully close the persistent event page after 3 hours of no user interaction with the computer, and the service worker background page should be able to receive this event for cleanup.
3. May not be guaranteed to be available on low power devices like mobile phones. Either the extension is not installable, or this information is provided when the extension tries to invoke the persistent event page.
4. User messaging informing the user that the extension is "active" somehow and may consume more resources

It should have an API so that the extension can turn it off whenever it wants.

---

This would completely address my use case. Can anyone else chime in if it would address theirs as well?

[bershanskiy]

Well, the whole point of MV3 is removing persistent pages... perhaps, browsers will allow event pages to persist while they use an active stream.

[wolfspelz]

1. Yes, if a tab with the extension's content script hat communicates with the background counts as "direct browser extension interaction".

BUT:

2. The browser should observer resource usage and kill extensions if they go beyond limits. Limits should be published and should depend on the platform: desktop, mobile.

3. Not on the phone is the status quo. But I am still hoping to convince people that a persistent background page that is only event driven is resource effective and can be on mobile as well.

4. Such messaging to the user is not good. Users are either frightened by these messages or ignore it. Shifts responsibility to the user when in fact it is our (developer) responsibility to make things right. Few extensions needing such messaging means users don't like them. Once they are used to it they don't care. Same with extension rights. A content script that writes to the DOM is marked as "can read all your data" even though it does not read data, just inserts a div. Some users are afraid that the extension reads their passwords, others always click OK as they learned from the phone app store. My impression "4. User messaging" is a compromise. Our and Google's responsibility is to protect users from resource misuse. The browser should measure extension CPU and react. Not tell users that things might be bad.

[aeharding]

@bershanskiy

perhaps, browsers will allow event pages to persist while they use an active stream.

That would partially solve my problem. However in my extension sometimes the tab isn't recorded (it's an option), but the tab is still "controlled" by a websocket that can stop it at any time.

Perhaps an event page would persist if a websocket was open as well. That would solve my problem.