iden3 / circomlibjs

Javascript library to work with circomlib circuits
74 stars 55 forks source link

Error on import inside worker module #22

Open fmerg opened 1 year ago

fmerg commented 1 year ago

I have a CPU intensive operation over JubJub, which I want to split into parallel tasks and offload to different worker threads. Each worker needs a eddsa instance to work with. My context does not allow for shared memory among workers and transferring eddsa to the worker threads is also not possible in any feasible way. I would like each worker to create its own eddsa instance:

const circomlibjs = require('circomlibjs');
...
const eddsa = await circomlibjs.buildEddsa();

However, importing circomlibjs inside the worker module raises an error when the worker starts running:

TypeError: Cannot destructure property 'mod' of 'threads.workerData' as it is undefined.

  at workerThread (node_modules/web-worker/cjs/node.js:151:5)
  at Object.<anonymous> (node_modules/web-worker/cjs/node.js:79:56)
  at Object.<anonymous> (node_modules/circomlibjs/node_modules/ffjavascript/build/main.cjs:8:14)

The exact error depends on the multithreading library in use. The above one is raised with workerpool. If I use the native node:worker_threads or web workers for the browser, I get something like

TypeError [ERR_INVALID_ARG_TYPE]: The "id" argument must be of type string. Received undefined
    at new NodeError (node:internal/errors:399:5)
    at validateString (node:internal/validators:163:11)
    at Module.require (node:internal/modules/cjs/loader:1120:3)
    at require (node:internal/modules/helpers:112:18)
    ...
    at Module.load (node:internal/modules/cjs/loader:1103:32) {
  code: 'ERR_INVALID_ARG_TYPE'

I do not face import issues with other libraries. I wonder if this is my fault (that is, if circomlibjs should be imported in a different way inside js threads) or a bug.

Any idea or hint is welcome. Thanks in advance.

tornadocontrib commented 3 months ago

@fmerg Solution would be requiring web-worker dependency. If you are bundling for the browser, you need to exclude it.

Something like

{
    input: 'src/merkleTreeWorker.ts',
    output: [
      {
        file: 'static/merkleTreeWorker.umd.js',
        format: "umd",
        esModule: false
      },
    ],
    treeshake: 'smallest',
    external: ['web-worker'],
    plugins: [
      esbuild({
        include: /\.[jt]sx?$/,
        minify: false,
        sourceMap: true,
        target: 'es2016',
      }),
      nodeResolve(),
      commonjs(),
      json(),
      replace({
        'process.browser': 'true'
      })
    ],
  }

Also look at the process.browser replace as the process object doesn't exist for browsers ( I think it is intended to support bundlers like webpack, but for rollup.js you need to manually replace it to true statement )

So either circomlibjs or ffjavascript would work inside Web Workers and node:worker_threads. Except that you need to bundle the worker file to use only either node.js or browser module.