Cannot find module in sveltekit project

lknlknim commented 10 months ago

Describe the bug The module cannot find it's internal files, should be vite related problem but not sure.

To Reproduce Steps to reproduce the behavior:

pnpm create svelte
Install @jsquash/png
Exclude optimization in vite.config.ts
import any @jsquash modules e.g. @jsquash/png
pnpm dev
See error

Reproduction URL https://codesandbox.io/p/github/lknlknim/sveltekit-jsquash/main

Expected behavior No error

Module Versions 2.1.4

Additional context The import was located in /src/routes/image/+server.ts

Tried both npm / yarn / pnpm, all giving the same error. Even copy the module folder to local folder won't work

What I was trying to achieve is to make it available on Cloudflare Pages functions(Cloudflare Worker) with Sveltekit to get rid of Cloudflare Image and Cloudflare Image Resize.

jamsinclair commented 10 months ago

This is because you're trying to run it Server Side in Node.js. As stated in the README, this package is built for the Browser. Node.js cannot import wasm files like the browser does 😿 .

There are much better image libraries already out there for Node.js that I can recommend:

Sharp - High performance Node.js image processing
Libsquoosh - From the creators of squoosh.app, it's no longer maintained but still works. Even production Next.js falls back to it.

If you want to dynamically process images you could try create a Cloudflare Worker with the Page Functions, however I am not familiar with that (Functions Documentation).

Best of luck! 🤞

lknlknim commented 10 months ago

Thanks for the quick response. I still got little confusion. So does it support V8 WITHOUT BROWSER? I was trying to do what you're saying. Sveltekit with svelte-adapter-cloudflare will bundle the whole website into Cloudflare Pages, and all server endpoints are extracted into single workers file aka Pages Functions / Cloudflare workers.

Do you mean this library ONLY not working in development because of Node.js, and it'll work in Cloudflare Workers in production? In fact I did tried Cloudflare Workers with wrangler dev in local and it's not working too. I'll try to deploy that and test.

jamsinclair commented 10 months ago

So does it support V8 WITHOUT BROWSER

I'll try and think of a way to rephrase this better in the README file. I meant it in regards to the restricted runtimes that serverless functions like Cloudflare Workers and Vercel Edge Functions use.

it'll work in Cloudflare Workers in production?

Potentially yes, you'll want to follow the instructions on how to support it in Cloudflare Workers.

Also... I just thought... you might be able to do it with Vite static imports 🤔 (I haven't tested this code, might need to tweak any mistakes I made)

import decode, { init } from '@jsquash/png/decode';
import PNG_WASM from '@jsquash/png/codec/squoosh_png_bg.wasm?url';

await init(PNG_WASM); 

// do what you need to do with the decoding.
const imageData = await decode(buffer);

lknlknim commented 10 months ago

I can't build with sveltekit project, the entry file can be found but the sibling files can't be found.

import { error } from '@sveltejs/kit';
import { dev } from '$app/environment';

import { IMAGE_ID_KEYWROD } from '$lib/config/keyword';
import { getObject } from '$lib/server/cloudflare/r2';
import { IMAGE_BUCKET_PATH } from '$lib/server/cms/image';

import decodeJpeg, { init as initJpegWasm } from '@jsquash/jpeg/decode.js';
import encodePng, { init as initPngWasm } from '@jsquash/png/encode.js';
import JPEG_DEC_WASM from '../../../node_modules/@jsquash/jpeg/codec/dec/mozjpeg_dec.wasm?url';
import PNG_DEC_WASM from '../../../node_modules/@jsquash/png/codec/squoosh_png_bg.wasm?url';

export function GET({ platform, url }) {
    const id = url.searchParams.get(IMAGE_ID_KEYWROD);
    if (!id) {
        throw error(412, 'Missing ID');
    }

    return getObject(platform!, `${IMAGE_BUCKET_PATH}/${id}`).then((object) => {
        if (!object) {
            throw error(404, 'Image not found');
        }

        if (dev) {
            return new Response(object.body);
        }

        return object
            .arrayBuffer()
            .then((buffer) =>
                Promise.all([initJpegWasm(JPEG_DEC_WASM), initPngWasm(PNG_DEC_WASM)])
                    .then(() => decodeJpeg(buffer))
                    .then((data) => encodePng(data))
            )
            .then((buffer) => new Response(buffer));
    });
}

import { sveltekit } from '@sveltejs/kit/vite';
import { defineConfig } from 'vite';
import mkcert from 'vite-plugin-mkcert';
import autoprefixer from 'autoprefixer';

export default defineConfig({
    server: { https: true },
    plugins: [sveltekit(), mkcert()],
    css: {
        postcss: {
            plugins: [autoprefixer()]
        }
    },
    optimizeDeps: {
        exclude: [
            '@jsquash/jxl',
            '@jsquash/avif',
            '@jsquash/webp',
            '@jsquash/png',
            '@jsquash/jpeg',
            '@jsquash/resize'
        ]
    },
    build: {
        rollupOptions: {
            plugins: [],
            external: [
                '@jsquash/jxl',
                '@jsquash/avif',
                '@jsquash/webp',
                '@jsquash/png',
                '@jsquash/jpeg',
                '@jsquash/resize'
            ]
        }
    }
});

node:internal/event_target:1083
  process.nextTick(() => { throw err; });
                           ^
Error [ERR_MODULE_NOT_FOUND]: Cannot find module 'C:\Projects\travel-suggest\node_modules\.pnpm\@jsquash+jpeg@1.2.0\node_modules\@jsquash\jpeg\utils' imported from C:\Projects\travel-suggest\node_modules\.pnpm\@jsquash+jpeg@1.2.0\node_modules\@jsquash\jpeg\decode.js
    at new NodeError (node:internal/errors:406:5)
    at finalizeResolution (node:internal/modules/esm/resolve:233:11)
    at moduleResolve (node:internal/modules/esm/resolve:845:10)
    at defaultResolve (node:internal/modules/esm/resolve:1043:11)
    at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:383:12)
    at ModuleLoader.resolve (node:internal/modules/esm/loader:352:25)
    at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:228:38)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:85:39)
    at link (node:internal/modules/esm/module_job:84:36)
Emitted 'error' event on Worker instance at:
    at [kOnErrorMessage] (node:internal/worker:326:10)
    at [kOnMessage] (node:internal/worker:337:37)
    at MessagePort.<anonymous> (node:internal/worker:232:57)
    at [nodejs.internal.kHybridDispatch] (node:internal/event_target:807:20)
    at exports.emitMessage (node:internal/per_context/messageport:23:28) {
  url: 'file:///C:/Projects/travel-suggest/node_modules/.pnpm/@jsquash+jpeg@1.2.0/node_modules/@jsquash/jpeg/utils',
  code: 'ERR_MODULE_NOT_FOUND'
}

I also can't build with the with-vite example, is there extra steps to do? I just git cloned it, pnpm i then pnpm build

PS C:\Projects\jSquash\examples\with-vite> pnpm build

> with-vite@0.0.0 build C:\Projects\jSquash\examples\with-vite
> vite build

vite v4.5.0 building for production...
transforming (34) node_modules\.pnpm\@jsquash+jxl@1.0.3\node_modules\@jsquash\jxl\codec\enc\jxl_enc.jsUnexpected early exit. This happens when Promises returned by plugins cannot resolve. Unfinished hook action(s) on exit:
(commonjs--resolver) resolveId "./codec/enc/avif_enc_mt" "C:/Projects/jSquash/examples/with-vite/node_modules/.pnpm/@jsquash+avif@1.1.2/node_modules/@jsquash/avif/encode.js"
(commonjs--resolver) resolveId "./codec/enc/jxl_enc_mt_simd" "C:/Projects/jSquash/examples/with-vite/node_modules/.pnpm/@jsquash+jxl@1.0.3/node_modules/@jsquash/jxl/encode.js"
(commonjs--resolver) resolveId "./codec/enc/jxl_enc_mt" "C:/Projects/jSquash/examples/with-vite/node_modules/.pnpm/@jsquash+jxl@1.0.3/node_modules/@jsquash/jxl/encode.js"
(vite:worker-import-meta-url) transform "C:/Projects/jSquash/examples/with-vite/node_modules/.pnpm/@jsquash+avif@1.1.2/node_modules/@jsquash/avif/codec/enc/avif_enc_mt.js"
(vite:worker-import-meta-url) transform "C:/Projects/jSquash/examples/with-vite/node_modules/.pnpm/@jsquash+jxl@1.0.3/node_modules/@jsquash/jxl/codec/enc/jxl_enc_mt.js"
(vite:worker-import-meta-url) transform "C:/Projects/jSquash/examples/with-vite/node_modules/.pnpm/@jsquash+jxl@1.0.3/node_modules/@jsquash/jxl/codec/enc/jxl_enc_mt_simd.js"
error during build:
Error: Unexpected early exit. This happens when Promises returned by plugins cannot resolve. Unfinished hook action(s) on exit:
(commonjs--resolver) resolveId "./codec/enc/avif_enc_mt" "C:/Projects/jSquash/examples/with-vite/node_modules/.pnpm/@jsquash+avif@1.1.2/node_modules/@jsquash/avif/encode.js"
(commonjs--resolver) resolveId "./codec/enc/jxl_enc_mt_simd" "C:/Projects/jSquash/examples/with-vite/node_modules/.pnpm/@jsquash+jxl@1.0.3/node_modules/@jsquash/jxl/encode.js"
(commonjs--resolver) resolveId "./codec/enc/jxl_enc_mt" "C:/Projects/jSquash/examples/with-vite/node_modules/.pnpm/@jsquash+jxl@1.0.3/node_modules/@jsquash/jxl/encode.js"
(vite:worker-import-meta-url) transform "C:/Projects/jSquash/examples/with-vite/node_modules/.pnpm/@jsquash+avif@1.1.2/node_modules/@jsquash/avif/codec/enc/avif_enc_mt.js"
(vite:worker-import-meta-url) transform "C:/Projects/jSquash/examples/with-vite/node_modules/.pnpm/@jsquash+jxl@1.0.3/node_modules/@jsquash/jxl/codec/enc/jxl_enc_mt.js"
(vite:worker-import-meta-url) transform "C:/Projects/jSquash/examples/with-vite/node_modules/.pnpm/@jsquash+jxl@1.0.3/node_modules/@jsquash/jxl/codec/enc/jxl_enc_mt_simd.js"
    at process.handleBeforeExit (file:///C:/Projects/jSquash/examples/with-vite/node_modules/.pnpm/rollup@3.29.4/node_modules/rollup/dist/es/shared/node-entry.js:25902:28)
    at Object.onceWrapper (node:events:629:26)
    at process.emit (node:events:514:28)
 ELIFECYCLE  Command failed with exit code 1.

jamsinclair commented 10 months ago

Thanks @lknlknim, alas, I don't think currently I can make this work for your use case.

You have encountered two problems from the provided code.

One is a bug in Vite (#37) that is causing builds to fail that use multi-threading.

The second is some issues I'd need to solve to support Node.js. I've documented them in #38, however, I can't guarantee any timeline where I can fix that. Perhaps there might be some other WebAssembly libraries out there that would work for your scenario?

lknlknim commented 10 months ago

I did some experiments and made it works on Cloudflare Workers, both local and production. Seems like what I was trying to do is impractical. Having both jpeg/png/webp/avif/hxl, decode/encode, resize into 1 Workers will bloat the bundle size to >6MB where Workers free tier is limited to <1MB. So even if we can build it with Sveltekit we still can't deploy to Cloudflare Pages free tier and even if it does it'll potentially extend the boot time. It's better to use with standalone Workers instead of gluing with web frameworks which is having tighter bundle size budget. With Cloudflare Pages/Workers we can use service binding to invoke standalone Workers without external request. And splitting this into 11 standalone Workers for each image format may do the tricks.

jamsinclair / jSquash

Cannot find module in sveltekit project #36