universal-ify the pre processing of workerize-loader?

[quentin-sommer]

This is more a discussion than an issue.

I'm building a worker-map library that takes a loop and splits it into several workers (following the map/reduce pattern). The downside right now is that all the worker code has to be passed as a string (because as @developit already said in a tweet export isn't valid syntax in a closure)

Here's the API at the moment.

import workerize from 'workerize'

async function workerMap(consumerString, data, {concurrency = 4} = {}) {
  const promisesArr = []
  const step = Math.ceil(data.length / concurrency)

  // map
  for (let i = 0; i < data.length; i += step) {
    const consumer = workerize(consumerString)
    // expecting a export consume(array, workerNb) function in the worker
    promisesArr.push(consumer.consume(data.slice(i, i + step), i / step + 1))
  }
  const ret = await Promise.all(promisesArr)

  // reduce
  const concated = ret.reduce((acc, cur) => acc.concat(cur), [])

  return concated
}

What do you think would be doable/a good idea to do? Is it even possible to extract the code from a module import?

Quentin

[developit]

Module import doesn't give access to the original code - sadly, there's no way to do that. If you're okay having exports in your worker, we could implement my suggestion to allow this:

workerMap(`export * from './actual-module'`)

[quentin-sommer]

Would this be processed by the bundler or dynamically resolved by workerize?

I'm also thinking in the end if one is using such a module to increase performance it does not make sense to add a lot of string manipulation on top of every call to spawn the actual worker

[developit]

Nope, that'd be leaving the browser to resolve the import.

Regarding performance - spawning a worker is an up-front cost, whereas calls to worker methods increase with usage. In the general case, very few workers are spawned but many methods are called.

[quentin-sommer]

Okay then sounds like a good workaround to me!

[janbaykara]

Had to deal with arbitrary multi-worker stuff for a project, and I ended up with this: https://gist.github.com/janbaykara/b9e8b50b00b705c58600c067eeca8cdd

[quentin-sommer]

Nice thanks for sharing!