Issues with multi-threaded worker modules and Vite

[jamsinclair]

When importing Oxipng, AVIF or JXL jSquash libraries with Vite the build fails to complete (Originally reported in #19)

These are upstream issues with Vite itself

• ~Build fails when nested workers are present (https://github.com/vitejs/vite/issues/13367)~ Since fixed in Vite >= v5.0.0-beta.11
• https://github.com/vitejs/vite/issues/7015

I don't think this will be solved anytime soon so I hope to work on #33 in the near future

[Kam1k4dze]

Hi, is there now any workaround to use AVIF with Vite or maybe alternatives?

[jamsinclair]

Hi @Kam1k4dze, yes I do have one potential work around.

As noted in the README I have released some special versions that you can install that should be compatible with Vite.

You should be able to install with:

npm install -S @jsquash/avif@1.1.2-single-thread-only

Please let me know if that doesn't work for you 🙇

[Kam1k4dze]

When I run vite build everything works fine, but when I run vite dev and try to import encode I get this error:

|- Error [ERR_MODULE_NOT_FOUND]: Cannot find module 'D:\VSCodeProjects\test\SortyPicks\node_modules\@jsquash\avif\encode' imported from D:\VSCodeProjects\test\SortyPicks\node_modules\@jsquash\avif\index.js
    at new NodeError (node:internal/errors:406:5)
    at finalizeResolution (node:internal/modules/esm/resolve:233:11)
    at moduleResolve (node:internal/modules/esm/resolve:845:10)
    at defaultResolve (node:internal/modules/esm/resolve:1043:11)
    at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:383:12)
    at ModuleLoader.resolve (node:internal/modules/esm/loader:352:25)
    at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:228:38)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:85:39)
    at link (node:internal/modules/esm/module_job:84:36)

When using version 1.3.0 vite dev works, but vite build gives an error related to the vite bug

[jamsinclair]

@Kam1k4dze Are you trying to run this code on the serverside with Vite SSR? Your error message looks to be showing node import resolutions.

If so, you should refactor your code to ensure the jSquash related code is only activated in the browser and is not executed during serverside rendering. Unfortunately, I am unsure with how this is done and you'll need to check the documentation or other information websites.

If you do need serverside rendering, you're likely better off either pre-processing your images in Node.js or other tools or using a Node.js based image library like Squoosh Node.js library and Sharp.

[Kam1k4dze]

Yeah, you're right, I forgot that ssr was enabled on that page to test it. Thank you very much for your help in resolving my issue!❤️

[jaskaransarkaria]

vite build --debug helped me get a find a workaround for this (for the avif and jxl modules at least). Vite can't resolve the location of the dependency causing a memory leak and a very long build that eventually crashes. The build gets into a infinite resolve loop with itself and ends in JavaScript heap out of memory error.

The @jsquash/avif@1.1.2-single-thread-only releases don't overcome this issue (in my experience).

Vite v5 build output

vite v5.0.3 building for production...
transforming (1) index.html  vite:resolve 0.30ms avif_enc_mt.worker.js -> /home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/avif/codec/enc/avif_enc_mt.worker.js +0ms
  vite:resolve 0.09ms avif_enc.wasm -> /home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/avif/codec/enc/avif_enc.wasm +0ms
  vite:resolve 0.06ms avif_dec.wasm -> /home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/avif/codec/dec/avif_dec.wasm +7ms
  vite:resolve 0.12ms mozjpeg_enc.wasm -> /home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/jpeg/codec/enc/mozjpeg_enc.wasm +11ms
  vite:resolve 0.06ms squoosh_png_bg.wasm -> /home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/png/codec/squoosh_png_bg.wasm +2ms
  vite:resolve 0.06ms mozjpeg_dec.wasm -> /home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/jpeg/codec/dec/mozjpeg_dec.wasm +7ms
transforming (19) node_modules/@jsquash/png/codec/squoosh_png.js  vite:resolve 0.36ms avif_enc_mt.worker.js -> /home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/avif/codec/enc/avif_enc_mt.worker.js +0ms
  vite:resolve 0.18ms avif_enc_mt.worker.js -> /home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/avif/codec/enc/avif_enc_mt.worker.js +0ms
  vite:resolve 0.20ms avif_enc_mt.worker.js -> /home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/avif/codec/enc/avif_enc_mt.worker.js +0ms
  vite:resolve 0.14ms avif_enc_mt.worker.js -> /home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/avif/codec/enc/avif_enc_mt.worker.js +0ms
  vite:resolve 0.12ms avif_enc_mt.worker.js -> /home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/avif/codec/enc/avif_enc_mt.worker.js +0ms
  vite:resolve 0.12ms avif_enc_mt.worker.js -> /home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/avif/codec/enc/avif_enc_mt.worker.js +0ms
  vite:resolve 0.15ms avif_enc_mt.worker.js -> /home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/avif/codec/enc/avif_enc_mt.worker.js +0ms
  vite:resolve 0.47ms avif_enc_mt.worker.js -> /home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/avif/codec/enc/avif_enc_mt.worker.js +0ms
  vite:resolve 0.13ms avif_enc_mt.worker.js -> /home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/avif/codec/enc/avif_enc_mt.worker.js +0ms
  vite:resolve 0.20ms avif_enc_mt.worker.js -> /home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/avif/codec/enc/avif_enc_mt.worker.js +0ms

...
...
...

<--- Last few GCs --->

[28991:0x611bce0]   105104 ms: Mark-sweep 4014.2 (4130.2) -> 4000.2 (4130.4) MB, 1169.3 / 0.0 ms  (average mu = 0.113, current mu = 0.021) task scavenge might not succeed
[28991:0x611bce0]   106314 ms: Mark-sweep 4016.6 (4131.0) -> 4002.9 (4131.4) MB, 1182.9 / 0.0 ms  (average mu = 0.069, current mu = 0.022) allocation failure scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb09980 node::Abort() [node]
 2: 0xa1c235 node::FatalError(char const*, char const*) [node]
 3: 0xcf784e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xcf7bc7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xeaf465  [node]
 6: 0xeaff46  [node]
 7: 0xebe46e  [node]
 8: 0xebeeb0 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xec1e2e v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
10: 0xe8336a v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
11: 0x11fc0b6 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
12: 0x15f0b19  [node]
zsh: IOT instruction  npm run build -- --debug

vite v4 output

vite v4.5.2 building for production...
  vite:esbuild 19.16ms tsconfck init /home/jaskaran/repos/jSquash +0ms
transforming (1) index.html  vite:resolve 0.26ms avif_enc.wasm -> /home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/avif/codec/enc/avif_enc.wasm +0ms
  vite:resolve 0.18ms avif_dec.wasm -> /home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/avif/codec/dec/avif_dec.wasm +12ms
  vite:resolve 0.18ms avif_enc_mt.worker.js -> /home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/avif/codec/enc/avif_enc_mt.worker.js +0ms
  vite:resolve 0.16ms mozjpeg_enc.wasm -> /home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/jpeg/codec/enc/mozjpeg_enc.wasm +31ms
  vite:resolve 0.15ms mozjpeg_dec.wasm -> /home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/jpeg/codec/dec/mozjpeg_dec.wasm +6ms
  vite:resolve 0.16ms jxl_enc_mt_simd.worker.js -> /home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/jxl/codec/enc/jxl_enc_mt_simd.worker.js +28ms
  vite:resolve 0.12ms jxl_enc_mt.worker.js -> /home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/jxl/codec/enc/jxl_enc_mt.worker.js +4ms
  vite:resolve 0.14ms jxl_enc.wasm -> /home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/jxl/codec/enc/jxl_enc.wasm +16ms
  vite:resolve 0.23ms squoosh_png_bg.wasm -> /home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/png/codec/squoosh_png_bg.wasm +1ms
  vite:resolve 0.15ms jxl_dec.wasm -> /home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/jxl/codec/dec/jxl_dec.wasm +7ms
transforming (27) node_modules/@jsquash/webp/meta.js  vite:resolve 0.21ms webp_enc_simd.wasm -> /home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/webp/codec/enc/webp_enc_simd.wasm +5ms
  vite:resolve 0.12ms webp_enc.wasm -> /home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/webp/codec/enc/webp_enc.wasm +3ms
  vite:resolve 0.18ms webp_dec.wasm -> /home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/webp/codec/dec/webp_dec.wasm +3ms
transforming (32) node_modules/@jsquash/webp/codec/enc/webp_enc_simd.js  vite:resolve 0.18ms avif_enc_mt.worker.js -> /home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/avif/codec/enc/avif_enc_mt.worker.js +0ms
transforming (38) node_modules/@jsquash/avif/codec/enc/avif_enc.jsUnexpected early exit. This happens when Promises returned by plugins cannot resolve. Unfinished hook action(s) on exit:
(commonjs--resolver) resolveId "./codec/enc/avif_enc_mt.js" "/home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/avif/encode.js"
(commonjs--resolver) resolveId "./codec/enc/jxl_enc_mt_simd.js" "/home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/jxl/encode.js"
(commonjs--resolver) resolveId "./codec/enc/jxl_enc_mt.js" "/home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/jxl/encode.js"
(vite:worker-import-meta-url) transform "/home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/avif/codec/enc/avif_enc_mt.js"
(vite:worker-import-meta-url) transform "/home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/jxl/codec/enc/jxl_enc_mt_simd.js"
(vite:worker-import-meta-url) transform "/home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/jxl/codec/enc/jxl_enc_mt.js"
error during build:
Error: Unexpected early exit. This happens when Promises returned by plugins cannot resolve. Unfinished hook action(s) on exit:
(commonjs--resolver) resolveId "./codec/enc/avif_enc_mt.js" "/home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/avif/encode.js"
(commonjs--resolver) resolveId "./codec/enc/jxl_enc_mt_simd.js" "/home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/jxl/encode.js"
(commonjs--resolver) resolveId "./codec/enc/jxl_enc_mt.js" "/home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/jxl/encode.js"
(vite:worker-import-meta-url) transform "/home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/avif/codec/enc/avif_enc_mt.js"
(vite:worker-import-meta-url) transform "/home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/jxl/codec/enc/jxl_enc_mt_simd.js"
(vite:worker-import-meta-url) transform "/home/jaskaran/repos/jSquash/examples/with-vite/node_modules/@jsquash/jxl/codec/enc/jxl_enc_mt.js"
    at process.handleBeforeExit (file:///home/jaskaran/repos/jSquash/examples/with-vite/node_modules/rollup/dist/es/shared/node-entry.js:25902:28)
    at Object.onceWrapper (node:events:646:26)
    at process.emit (node:events:526:28)

This workaround works if you don't need to use the webworker, because the workaround ignores building them and works with vite v4 and vite v5:

import { defineConfig } from 'vite'

// https://github.com/vitejs/vite/blob/ec7ee22cf15bed05a6c55693ecbac27cfd615118/packages/vite/src/node/plugins/workerImportMetaUrl.ts#L127-L128
const workerImportMetaUrlRE =
  /\bnew\s+(?:Worker|SharedWorker)\s*\(\s*(new\s+URL\s*\(\s*('[^']+'|"[^"]+"|`[^`]+`)\s*,\s*import\.meta\.url\s*\))/g

export default defineConfig({
  optimizeDeps: {
    exclude: ["@jsquash/avif", "@jsquash/jpeg", "@jsquash/jxl", "@jsquash/png", "@jsquash/webp"]
  },
  worker: {
    format: 'es',
    // https://github.com/vitejs/vite/issues/7015
    // https://github.com/vitejs/vite/issues/14499#issuecomment-1740267849
    plugins: [
      {
        name: 'Disable nested workers',
        enforce: 'pre',
        transform(code, id) {
          if (code.includes('new Worker') && code.includes('new URL') && code.includes('import.meta.url')) {
            return code.replace(workerImportMetaUrlRE, `((() => { throw new Error('Nested workers are disabled') })()`);
          }
        }
      }
    ]
  }
})

Credit to @sapphi-red for providing the workaround. Also, huge thanks to @jamsinclair for creating the module, it's really fantastic work 🎉

[jamsinclair]

The @jsquash/avif@1.1.2-single-thread-only releases don't overcome this issue (in my experience).

@jaskaransarkaria I think you may have not installed the package correctly. Your stacktrace shows the file avif_enc_mt.worker.js which is not included in the single threaded build.

[jamsinclair]

It seems like from Vite v5.2.2 this issue should be resolved.

The following Vite config may need to be added:

worker: {
    format: "es"
}

I'll update this issue once fully confirmed.

[tasiotas]

I confirm that now I can successfully build project with "@jsquash/avif": "1.3.0" and vite

[Ice-Hazymoon]

hi, @jamsinclair, I found that the @jsquash/oxipng package still doesn't build properly with Vite, while other packages are fine.

[Ice-Hazymoon]

hi, @jamsinclair, I found that the @jsquash/oxipng package still doesn't build properly with Vite, while other packages are fine.

Nuxt Build Error: [commonjs--resolver] Circular worker imports detected. Vite does not support it. Import chain: node_modules/.pnpm/@jsquash+oxipng@2.2.0/node_modules/@jsquash/oxipng/codec/pkg-parallel/snippets/wasm-bindgen-rayon-7afa899f36665473/src/workerHelpers.js -> node_modules/.pnpm/@jsquash+oxipng@2.2.0/node_modules/@jsquash/oxipng/codec/pkg-parallel/snippets/wasm-bindgen-rayon-7afa899f36665473/src/workerHelpers.js

[jamsinclair]

Thanks @Ice-Hazymoon. Thanks to updates in https://github.com/RReverser/wasm-bindgen-rayon we can now support Vite.

I'll need to compile a new version of the oxipng codec and publish. I'll comment once completed 👍

[jamsinclair]

@Ice-Hazymoon I've published a new version of @jsquash/oxipng, v2.3.0.

Please let me know if that solves the issue 🙇 . Happy to help further, if not.

The Vite dependency will need to be greater than v5.1.6. You may also need to add the following config to vite.config.js.

worker: {
    format: "es"
}