sveltejs / kit

web development, streamlined
https://svelte.dev/docs/kit
MIT License
18.64k stars 1.92k forks source link

`esbuild` causes import errors with `adapter-node` #6440

Closed AlexRMU closed 1 year ago

AlexRMU commented 2 years ago

Describe the bug

Same issue as https://github.com/sveltejs/kit/issues/2400

After https://github.com/sveltejs/kit/pull/6372, I get:

var __require = /* @__PURE__ */ ((x) => typeof require !== "undefined" ? require : typeof Proxy !== "undefined" ? new Proxy(x, {
    get: (a, b) => (typeof require !== "undefined" ? require : a)[b]
}) : x)(function (x) {
    if (typeof require !== "undefined")
        return require.apply(this, arguments);
    throw new Error('Dynamic require of "' + x + '" is not supported');
});

...

var crypto_1 = __importDefault(__require("crypto"));

Perhaps the problem is https://github.com/evanw/esbuild/issues/1921

Reproduction

https://stackblitz.com/edit/sveltejs-kit-template-default-sczcqj?file=README.md

Logs

No response

System Info

any

Severity

blocking all usage of SvelteKit

Additional Information

No response

Rich-Harris commented 2 years ago

This is happening because we now bundle your devDependencies, so that they don't need to be installed alongside your deployment.

Your app contains some dependencies that can't be bundled. I was able to get your repro to build and run by moving mongoose and html-minifier from devDependencies to dependencies.

t-heuser commented 2 years ago

Having the same issue after upgrading to the newest versions of kit and adapter-node. But in my case it says Error: Dynamic require of "tty" is not supported when trying to run in production mode. So unfortunately I cannot just move any devDependency to dependency to fix the issue. Or am I missing something here?

AlexRMU commented 2 years ago

How do I find out which dependencies can't be bundled? Just because such errors appear? For example, if I build only mongoose and add require, everything will work: https://stackblitz.com/edit/sveltejs-kit-template-default-gikzvb?file=README.md And uglify-js contains the following code:

exports.FILES = [
    require.resolve("../lib/utils.js"),
    ...
];

which for some reason cannot be bundled at the moment. Maybe esbuild needs to add a shim for require.resolve (https://www.npmjs.com/package/@chialab/esbuild-plugin-require-resolve, https://github.com/evanw/esbuild/issues/1311#issuecomment-849144027).

AlexRMU commented 2 years ago

@oneserv-heuser Can you show your vite.config.js?

t-heuser commented 2 years ago

@AlexRMU Yes, i could but there is nothing spectacular to see, just adds the sveltekit plugin 😀

Solved my issue, I referenced my tailwindconfig in my src code which has led to the bundling of tailwindcss and all its dependencies in the production build. The module picocolors (dependency of tailwindcss) needed tty which led to the error. Removed the import of values from the tailwindconfig and everything is working now.

AlexRMU commented 2 years ago

Without mongoose and html-minifier, the same Dynamic require appears. Solved by adding require.

Rich-Harris commented 2 years ago

Solved by adding require.

What does this mean?

ctiospl commented 2 years ago

Having a similar issue after updating to adapter-node 1.0.0-next.88 Error: Dynamic require of "fs" is not supported not sure if its got to do with ant devdependencies "devDependencies": { "@sveltejs/adapter-auto": "1.0.0-next.71", "@sveltejs/adapter-node": "1.0.0-next.88", "@sveltejs/kit": "1.0.0-next.460", "@tailwindcss/forms": "^0.5.2", "@tailwindcss/line-clamp": "^0.4.0", "@tailwindcss/typography": "^0.5.4", "@types/cookie": "^0.5.1", "@typescript-eslint/parser": "^5.36.1", "autoprefixer": "^10.4.8", "dotenv": "^16.0.2", "env-cmd": "^10.1.0", "eslint": "^8.23.0", "postcss": "^8.4.16", "postcss-load-config": "^4.0.1", "prettier": "^2.7.1", "prettier-plugin-svelte": "^2.7.0", "svelte": "^3.49.0", "svelte-check": "^2.9.0", "svelte-drawer-component": "^1.2.2", "svelte-popperjs": "^1.3.2", "svelte-preprocess": "^4.10.7", "svelte-table": "^0.5.0", "tailwindcss": "^3.1.8", "tslib": "^2.4.0", "type-fest": "^2.19.0", "typescript": "^4.8.2", "vite": "3.1.0-beta.1" }, "dependencies": { "@fontsource/fira-mono": "^4.5.9", "@popperjs/core": "^2.11.6", "@rgossiaux/svelte-headlessui": "^1.0.2", "@supercharge/fs": "^3.4.0", "cookie": "^0.5.0", "daisyui": "^2.24.0", "date-fns": "^2.29.2", "date-fns-tz": "^1.3.7", "deep-object-diff": "^1.1.7", "knex": "^2.3.0", "mysql": "^2.18.1", "njwt": "^1.2.0", "svelecte": "^3.9.2", "svelte-command-palette": "^1.2.0" }

AlexRMU commented 2 years ago

@Rich-Harris As it is written in the readme in the reproduction, you need to add

import { createRequire } from "module";
const require = createRequire(import.meta.url);

to the build before var __require = ...

AlexRMU commented 2 years ago

@chintan-infiniteshopping If you move all devDependencies to dependencies, this error will not occur, but the dependencies will not be bundled and they will need to be installed before running

spheenik commented 2 years ago

@Rich-Harris

Solved by adding require.

What does this mean?

I am wrestling with this problem for quite a while now. The code that @AlexRMU mentioned is in a chunk file, in my case build/server/chunk-BHN6OJC3.js

var __require = /* @__PURE__ */ ((x) => typeof require !== "undefined" ? require : typeof Proxy !== "undefined" ? new Proxy(x, {
    get: (a, b) => (typeof require !== "undefined" ? require : a)[b]
}) : x)(function (x) {
    if (typeof require !== "undefined")
        return require.apply(this, arguments);
    throw new Error('Dynamic require of "' + x + '" is not supported');
});

If you manually patch this file, and add the following directly above the above mentioned code

import { createRequire } from "module";
const require = createRequire(import.meta.url);

then this issue seems to be solved (it is on my side). Seems to be difficult to find an elegant solution though. That code is generated by esbuild. I tried to patch the adapter-node to by using esbuilds's banner configuration, that however adds the banner to every generated chunk, not just the one. Maybe there's some other nice fix.

I added the banner like explained here: https://github.com/evanw/esbuild/issues/1921#issuecomment-1152991694

dlebech commented 2 years ago

From @spheenik in https://github.com/sveltejs/kit/issues/6440#issuecomment-1236823838

If you manually patch this file, and add the following directly above the above mentioned code

import { createRequire } from "module";
const require = createRequire(import.meta.url);

then this issue seems to be solved (it is on my side).

I can confirm this fixed the issue in my build. I happen to have a chunk with exactly the same name server/chunk-BHN6OJC3.js (probably not a coincidence?), and adding the above code made the build work with node build afterwards.

My particular case:

Downgrading to adapter node next.87 for now solves the issue.

Anyass3 commented 2 years ago

I had a similar issue too. Thanks @dlebech

Downgrading to adapter node next.87 for now solves the issue.

This worked for me

theetrain commented 2 years ago

:eyes: Update 1 (2022 September 12): added Undici snippet


I added the banner like explained here: https://github.com/evanw/esbuild/issues/1921#issuecomment-1152991694 — @spheenik

I'm facing issues as well running:

👉 TL;DR: SvelteKit's dependencies are not being internalized as part of the built chunks.

My node build runtime error is:

file:///Users/me/repos/project/app/build/server/chunk-RFO2XWEP.js:13
  throw new Error('Dynamic require of "' + x + '" is not supported');
        ^

Error: Dynamic require of "assert" is not supported

:eyes: Update 1: And the stack trace points to this chunk snippet:

// node_modules/undici/lib/client.js
var require_client = __commonJS({
  "node_modules/undici/lib/client.js"(exports, module) {
    "use strict";
    var assert = __require("assert");

I'm still trying to figure this out, but I wonder if the polyfills in packages/kit/src/exports/node/polyfills.js are not being bundled correctly under certain conditions. I attempted to create a repository to reproduce the issue, making use of hooks and fetch in server-side load, but I cannot seem to replicate the issue found in my main project.


I tried adding the linked esbuild setting to my vite.config.js:

import { sveltekit } from '@sveltejs/kit/vite'

const config = {
  plugins: [sveltekit(),
  esbuild: {
    bundle: true,
    minify: true,
    format: 'esm',
    target: 'esnext',
    platform: 'node',
    banner: {
      js: 'import { createRequire } from "module";const require = createRequire(import.meta.url);'
    },
    outExtension: {
      '.js': '.mjs'
    }
  }
}

export default config

But then I get a build error (npm run build):

"banner" must be a string

Setting banner as a string banner: 'import { createRequire } from "module";const require = createRequire(import.meta.url);' yields a different error:

Invalid option in transform() call: "bundle"

Manually adding the desired banner string (adding createRequire) to the top of my errored chunk does resolve the issue, but I can't get esbuild to add that for me.

Downgrading to @sveltejs/adapter-node@1.0.0-next.87 doesn't resolve the issue for me since it displays cookie as an external dependency; and then it's a game of whack-a-mole to add the dozens of sub-dependencies that are not declared as devDependencies. For what it's worth, here are some extra details about my app's package.json:

My package.json ```json { "name": "app", "version": "0.0.1", "private": true, "scripts": { "build": "vite build", "check:watch": "svelte-check --tsconfig ./jsconfig.json --watch", "check": "svelte-check --tsconfig ./jsconfig.json", "package": "svelte-kit package", "preview": "vite preview --host", "start": "vite dev --host" }, "devDependencies": { "@sveltejs/adapter-node": "^1.0.0-next.87", "@sveltejs/kit": "^1.0.0-next.454", "carbon-components-svelte": "^0.70.1", "carbon-icons-svelte": "^11.1.0", "carbon-preprocess-svelte": "^0.9.1", "svelte": "^3.44.0", "svelte-progress-bar": "^3.0.2" }, "type": "module", "dependencies": { "mysql2": "^2.3.3", "slug": "^8.0.0" } } ``` ``` app % npm ls cookie app@0.0.1 /[redacted]/[path] └─┬ @sveltejs/kit@1.0.0-next.480 └── cookie@0.5.0 ```

What vite/esbuild config worked for you?

dlebech commented 2 years ago

@theetrain

What vite/esbuild config worked for you?

In my case, we're still on sveltekit next.471 and adapter node next.87, i.e. before the introduction of esbuild for including dependencies.

theetrain commented 2 years ago

@dlebech thanks for the suggestion; I went with the workaround you suggested and downgraded to node-adapter 87. Since my project is built as a docker image, I'll have to include all dependencies instead of the leaner production serve npm i --production && node build until this is resolved upstream.

I can tell that node-adapter is marking its dependencies as external:

https://github.com/sveltejs/kit/blob/70e3c34928b4acf408770a0a3bbce08081a92073/packages/adapter-node/index.js#L48-L60

Perhaps it needs to be bundled in, or all dependencies' dependencies should be recurisvely bundled; I'm not too sure.

ctiospl commented 2 years ago

Error: Dynamic require of "fs" is not supported

got this error resolved by using $env/static/private instead of dotenv for environment variables.

icalvin102 commented 2 years ago

@oneserv-heuser can you elaborate on how you solved the issue with tailwind? I'm not sure if I understand your solution correctly.

I'm also running into the Dynamic require of "tty" is not supported error that is caused by tailwind > picocolor. I never imported tailwind.config.cjs from any of my project files though.

theetrain commented 2 years ago

I updated my post above (https://github.com/sveltejs/kit/issues/6440#issuecomment-1242126133), noting a require error with the bundled unidici module calling __require("assert").

Here's a work in progress reproduction: https://github.com/theetrain/sveltekit-issue-6440 except unidici won't bundle with npm run build. In my main project, it does. While I try to reproduce the issue, can anyone lend me a hint on how undici would or could get bundled? I'm guessing it has to do with how these polyfills get bundled.

dlebech commented 2 years ago

@theetrain Quick note, your reproduction repository looks like it has some issues, for example you're trying to access fetch in page.server.js and layout.server.js from the server load event, but fetch is not available in server load events -- so it gives a warning and eventually leads to a runtime error "fetch is not a function", even when running the dev server (I'm on Node 16.15)

dlebech commented 2 years ago

Edit: I figured out why external dependencies were not being respected in the esbuild. I was importing firebase as @firebase/auth instead of firebase/auth, and I guess esbuild couldn't figure out that the @ import was the same. Changing my imports to firebase/X (without the @) seems to have resolved the issue for me, if I'm very diligent about putting libraries needed for the server in the dependencies list.


@Rich-Harris it's not enough to move dependencies from devDependencies to dependencies. Edit: It might be, see above.

Here's a reproduction of the issue with a single dependency firebase and it doesn't matter if it's included in dependencies or devDependencies: https://github.com/dlebech/sveltekit-6440

The manual workaround mentioned above solves the issue in this repo as well, but is not feasible to include in automated pipelines.

The banner trick also doesn't work, but I assume it's because adapter-node esbuild config is hardcoded and doesn't accept any config parameters.

t-heuser commented 2 years ago

@oneserv-heuser can you elaborate on how you solved the issue with tailwind? I'm not sure if I understand your solution correctly.

I'm also running into the Dynamic require of "tty" is not supported error that is caused by tailwind > picocolor. I never imported tailwind.config.cjs from any of my project files though.

I got the issue when using import tailwindConfig from '$lib/tailwind.config'. I used variables from the config file in my source code, e.g. passing colors to libraries to not hardcode colors in thge source code again, when all colors are configured in the tailwind config file. Normally the tailwindconfig resides in the root, but vite won't bundle it then correctly so I had to move it to the lib directory. Importing something from the tailwindconfig caused the bundling of the entire tailwind package and all of it's dependencies in the production code as in my tailwindconfig I imported all the colors like this: import colors from 'tailwindcss/colors.js'. This then led to the error as described because of the usage of tty in the browser which is not possible.

I solved the issue with moving the tailwindconfig back to where it belongs, to the root directory of my project. I then wrote a little rollup plugin which takes the config file, extracts all the parameters I use in my production code and then writes it into a new file residing in src/lib. So whenever the applicaton is rebuild I get a minimum file with all the values I need without the import of anything from tailwind as everything gets resolved in the build step.

import tailwindConfig from '../tailwind.config.js'
import * as fs from 'fs'
import { pick } from 'lodash-es'

/* All keys which are needed in the application from the tailwind configuration. */
const requiredKeys = ['screens', 'colors', 'extend', 'spacing']

/**
 * This takes the tailwind config file, prepares it and writes it to src/lib so values from the configuration can be
 * used in components.
 */
export default function bundleTailwindConfig () {
  return {
    name: 'tailwind-config-bundler',
    buildStart () {
      let configObject = tailwindConfig.theme
      configObject = pick(configObject, requiredKeys)
      const content = `/* this file is generated — do not edit it! */ export default ${JSON.stringify(configObject)}`
      fs.writeFile('./src/lib/tailwind.config.js', content, 'utf8', (error) => {
        if (error) throw error

        console.log('Tailwind config file successfully processed.')
      })
    }
  }
}

Hope this explanation is sufficient. But I cant help you with the issue that you get the same error as me without importing anything from tailwind :/

AlexRMU commented 2 years ago

So, https://github.com/sveltejs/kit/pull/6896 appeared and I think the problem with Dynamic require is solved.


When I try to build all the dependencies, it appears: A lot of Circular dependency: ... and

<--- Last few GCs --->

[7236:000002396E15EFC0]   108147 ms: Scavenge 4009.4 (4096.6) -> 4008.8 (4107.3) MB, 6.4 / 0.0 ms  (average mu = 0.338, current mu = 0.329) allocation failure
...

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 00007FF6FC137A1F v8::internal::CodeObjectRegistry::~CodeObjectRegistry+114207
...

It was solved by changing the max-old-space-size, but this is not always possible. Apparently, this is not a rollup problem, you need to wait for ebuild to solve their problems.

theetrain commented 2 years ago

So, https://github.com/sveltejs/kit/pull/6896 appeared and I think the problem with Dynamic require is solved. — @AlexRMU

Same here; after upgrading to @sveltejs/adapter-node@1.0.0-next.95 my issue above (https://github.com/sveltejs/kit/issues/6440#issuecomment-1242282652) went away for me! 🎉

I can now run a built application after installing production modules only via npm i --production && node build.

bluwy commented 2 years ago

adapter-node now uses rollup (#6896) which I believe solved the issue here

swyxio commented 2 years ago

@bluwy btw i am also seeing this in adapter-auto on Netlify as well

https://github.com/sw-yx/swyxkit/pull/117

image

i am not sure how to fix.

bluwy commented 2 years ago

@sw-yx Seems like it's happening for adapter-netlify (which adapter-auto picks) which is still using esbuild. But the original cause #6372 only changes adapter-node, so something else might have caused it between v480 to v505. I'm not sure of a workaround too.

swyxio commented 2 years ago

agree. i'll just pin my version for adapter-netlify right now, but this continuing to be a bug/problem for netlify as far as i'm aware.

cc @brittneypostma fyi

swyxio commented 2 years ago

issue dates back to adapter auto v74 as far as i can tell https://github.com/sw-yx/swyxkit/pull/117; last working version is v72

swyxio commented 1 year ago

I am still seeing this issue today, preventing any upgrade of sveltekit adapter netlify version (current version 84) https://github.com/sveltejs/kit/issues/7839

benmccann commented 1 year ago

adapter-node doesn't use esbuild currently, so I'm going to close this in favor of the issue you just filed

danez commented 1 year ago

We are currently also rolling out a fix that should fix the issue in the bundling on Netlify when using esbuild as bundler.

swyxio commented 1 year ago

ah wonderful, thank you!