Inlining workers

[zeusdeux]

Hi folks!

Would there be concerns regarding adding support for inlining workers using a plug-in such as rollup-plugin-web-worker-loader for example? If not, I could PR support for it.

Our use case is that we have a bunch of internal libraries that are meant for the browser which ship with web workers that offload heavy tasks from the main thread. Our current tooling of choice has aged poorly and we are currently using microbundle as the new bundler for libraries.

Support for inlining would help us cover all our use cases and thus push for adoption.

Thank you!

[zeusdeux]

I see https://github.com/developit/microbundle/issues/170 but for some reason I don't see the base64 bundled worker in my build.

Is this because OMT changed in someway or should I post a reproduction repo here?

[ForsakenHarmony]

I'm guessing you've gone through the ways of using OMT https://github.com/surma/rollup-plugin-off-main-thread#auto-bundling?

You're welcome to make a PR to fix worker support

[zeusdeux]

Hi @ForsakenHarmony! I sure have but none of them result in a bundled worker unfortunately hence my initial query. Tried the omt:... import syntax as well but to no avail.

I would be down to PR a fix. Does that mean we are confirming an issue with microbundle then?

[developit]

@zeusdeux hmm - I was hesitant to add support for bundling workers to inline strings, because that's generally pretty bad for performance. Do you have a motivating use-case for consideration?

Currently, Microbundle will bundle workers as entrypoints and link them via new Worker(new URL('./bundle.js',import.meta.url),{type:'module'}) so they get picked up by most bundlers.

[zeusdeux]

Hey @developit! The core of our use case stems from the fact that we author everything in TypeScript and all our libraries are co-located in a massive monorepo which is tested using Jest. All this tooling is abstracted out in the root of the repo and shared by all packages in the repo.

Using the new URL syntax (or the deprecated string path for that matter) with a worker written in TS leaves the worker code and invocation site as is and it doesn't get transpiled, hashed named and have the path in source re-written when using the --workers option to generate a bundle from the source prop.

We work around this by running the worker through microbundle first and writing the transpiled version to a stable location in dist. This stable location is what we manually leave in the new Worker call. Our understanding was that with this, microbundle would see that the path in new Worker is to a js file, roll it up and inline it in the main and module bundles. Since that currently isn't the case, all the consumers of these libraries have loaders to lift these workers out and put them in a folder that is then pushed to the CDN. Which, to me, feels like all they are really doing is a bunch of mvs in a situation that should be transparent for them.

This results in a maintenance hurdle as it causes us to own tiny build parts of many disparate codebases that we don't have expertise on.

Given that our workers are fairly small and we do a lot of work around only loading what we need when we need it, inlining them isn't a performance issue from our current measurements. The benefits we receive in the form of maintenance and iteration speed is quite high.

It also makes it easy for us to pull out other bits of code that our team has expertise on, from our existing apps and pull them into the library monorepo if we have this bundling infrastructure in place.

With these constraints, the capability to inline workers would be very beneficial in our efforts.

I am happy to have a conversation around this and happy to lend a hand where we can! 😊

(Please ignore the typos, doing this one the phone 😅)

PS: import.meta.url necessitated by new URL currently breaks Jest as their experimental esm implementation is, well, experimental. So we are stuck with string path and then webpack magic in app build infrastructure.