w3c / webextensions

Charter and administrivia for the WebExtensions Community Group (WECG)
Other
578 stars 50 forks source link

Challenging Service Worker Use Case - Machine Learning Models #73

Open wingman-jr-addon opened 2 years ago

wingman-jr-addon commented 2 years ago

This a use case in response to the request for challenging Service Worker scenarios in #72 .

Machine learning is at an inflection point where we are starting to see models deployed in a variety of ways. I write an addon for Firefox that uses machine learning, and have been involved in the community for a similar one that is cross-browser. Both of these leverage Tensorflow.js, a common choice for this type of task. Loading models (including first inference warmup) takes time on the order of seconds - even for small models that are quite reasonable to deploy like MobileNetV2. While this may be somewhat dependent on the library implementation and the model, experience with native libraries leads me to believe that this load time can only be optimized so far.

The lifecycle aspect of Service Workers makes this long load time challenging. (Aside: I would also like to ensure that interaction with GPU resources be well-defined in the context of Service Workers.)

While it seems that an infinite lifetime might solve this, stability is always a competing concern for long-lived processes. I recently implemented a "watchdog" feature to detect a scenario where after some amount of time, model predictions were coming back with incorrect results. If the check failed, the plugin reloads. To this end, I think some notion of a finite lifetime could potentially be beneficial but how exactly the addon and the browser interact to accomplish this likely needs to be cooperative and preferably treated more like a maintenance window.

(For correlation purposes, please note I additionally go by the handle zephyr in the WECG chat.)

cuylerstuwe commented 2 years ago

For the time being, it seems that the only real solution for this would be to create a visible "fake" background.js file that tells the user something to the effect of "keep me open and minimize me, I'm doing work".

This is how I'm thinking I'm going to have to start implementing freelance client requests, given that they often need long-running functionality within an extension that's supported for the forseeable future. I'm wondering whether it will become a pattern amongst the extension development community.

Maybe the simplest solution browser implementors could provide to help developers with this problem would be to just give developers the capability of creating an invisible window (rather than forcing developers to create and minimize one that we beg our users not to close).

Once the idealism wears off and pragmatism sets in, we could possibly get something like:

chrome.windows.create({type: "hidden"}).

If browser vendors think this is a security and/or performance issue, perhaps they could simply surface the ability for users to see these hidden windows, e.g., if right-clicking on an extension's icon. Maybe it could also involve its own manifest permission (e.g., "invisibleWindows") with its own warning, e.g., "This extension uses long-running background processes, which could affect performance and battery life" etc. User-centric companies don't try to limit what users can opt into; They merely try to help users make informed decisions.

P.S., What's kinda interesting is that this sort of behavior is actually available in Chrome on OSX, possibly by accident. A window created with chrome.windows.create({type: "popup", state: "minimized"}) is not visible and cannot be opened. It's effectively the same as what I'm proposing with chrome.windows.create({type: "hidden"}). This seemingly-accidental feature doesn't really seem to be doing any harm. It's just too bad that it doesn't behave the same across browsers and operating systems.

wingman-jr-addon commented 2 years ago

@cuylerstuwe Fascinating - I ran into an issue with GPU support in background.js in Firefox, and ended up doing basically exactly the solution you describe using "hidden tabs" and implementing a client/server architecture to work around that issue. I wrote up a reflection on the transformation refactor if you're interested. I'd be curious your further perspectives on the long-term impacts of this as well for both developer and user.

wingman-jr-addon commented 2 years ago

As a followup note here, I collect feedback from departing users, and at least one specifically listed the annoying "hidden tabs" prompt - essentially the main viable workaround here - as a reason for leaving. So I can concretely say that the current workarounds available are known to cause user frustration.