Buffer is not defined - Githubissues

konsumer commented 2 years ago

What version of Remix are you using?

1.3.5

Steps to Reproduce

setup a remix cloudflare-worker project with mantine or any other library that uses Buffer. I followed the remix directions on the mantine site.
remix build: works fine
wrangler publish: fails

I can make a demo project that fails, if you want to try to deploy it to your own cloudflare.

Expected Behavior

Should output code that can be deployed on cloudflare.

Actual Behavior

It was reported at #2625 but said to be fixed in 1.3.5, so it may be some combination of remix/mantine/etc. I can build in remix fine, but on deploy (via wrangler), I get this:

✘ [ERROR] Received a bad response from the API

  Uncaught ReferenceError: Buffer is not defined
    at line 4699
    at line 33
    at line 5472
    at line 33
    at line 6626
    at line 33
    at line 6704
    at line 33
    at line 94195
   [code: 10021]

  If you think this is a bug, please open an issue at:
  https://github.com/cloudflare/wrangler2/issues/new

Normally, in other builders, a polyfill can be used, either in esbuild config, which we can't edit in remix, or by importing buffer. If I install buffer module & and this in server entry-point:

/* global addEventListener  */

import buffer from 'buffer'

import { createEventHandler } from '@remix-run/cloudflare-workers'
import * as build from '@remix-run/dev/server-build'

globalThis.Buffer = buffer

addEventListener(
  'fetch',
  createEventHandler({ build, mode: process.env.NODE_ENV })
)

I get this error:

Building Remix app in production mode...
It appears you're using a module that is built in to node, but you installed it as a dependency which could cause problems. Please remove buffer before continuing.
Error
    at Object.onBuildFailure (/Users/konsumer/Documents/work/PROJECT/node_modules/@remix-run/dev/cli/commands.js:150:13)
    at buildEverything (/Users/konsumer/Documents/work/PROJECT/node_modules/@remix-run/dev/compiler.js:280:13)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Object.build (/Users/konsumer/Documents/work/PROJECT/node_modules/@remix-run/dev/compiler.js:105:3)
    at async Object.build (/Users/konsumer/Documents/work/PROJECT/node_modules/@remix-run/dev/cli/commands.js:145:3)
    at async Object.run (/Users/konsumer/Documents/work/PROJECT/node_modules/@remix-run/dev/cli/run.js:467:7)

konsumer commented 2 years ago

Here is my related mantine issue. I am having a bit of trouble figuring out who to report it to, or where I could even work on a PR to fix the problem.

I made a demo project here, but couldn't even get it started using everyone's "getting started" directions, and remix-esbuild-override stopped working (which used to be how I solved this.)

konsumer commented 2 years ago

Related Discussion

3x071c commented 2 years ago

Cloudflare runs on V8 natively, without the additional Node.js abstraction. This means certain APIs, such as Buffer, aren't available for your dependencies to use in Cloudflare Workers/Pages. Try this, it allows you to override Remix's esbuild configuration to inject/alias polyfills. Here's an example on how to replace the Buffer API. If you want a full production usage example, check out this repository. I agree that Remix should be polyfilling Buffer by default on CF though, it's definitely a hurdle for adoption. (I think there already is an issue about that)

konsumer commented 2 years ago

Cloudflare runs on V8 natively, without the additional Node.js abstraction.

Yep, I'm aware. The way I generally solve this, in non-remix worker projects (I have quite a few) is to polyfill it in my esbuild (or whatever bundler I'm using) config, as I said above.

Try this, it allows you to override Remix's esbuild configuration to inject/alias polyfills.

I was originally using remix-esbuild-override but it stopped working in my project (as I mentioned above.)

I made this issue because I was told in discord that custom esbuild config would not be supported, but we could add issues for specific things I am having trouble with, in a case-by-case fashion. I'm not really here for helping/troubleshooting remix, or whatever, this is more for future-devs that have the same issue, which I imagine is a pretty widespread problem. If it's not helpful for me to just report the issue, like it doesn't seem important that users want to be able to fix their builds (with custom esbuild config) or more specifically, use any node library that uses Buffer, then feel free to close and mark "won't fix".

On my own project, I ended up just dropping remix. I could probably take the time to sort through what exact combo of versions of things started breaking, but honestly, remix is doing very little for me that I can't just do with a standard edge-worker and something that can build a static site (I am using vite, but also next static build works great, etc.) Our team is pretty comfortable with graphql and other react SSR frameworks, and edge-workers, so I kept feeling like we had to relearn things in a jenky way, that we could already do much better, the regular way. If I really needed SSR on this particular project, I might be more motivated to play around more with remix-esbuild-override, but it felt extremely flakey (often it would not correctly override, then just stopped working) and remix is just not needed in my current project. I had the whole thing rebuilt and working without remix, in about a day, and I will be hesitant to use or recommend remix, in the future.

My current setup is incredibly reliable, and all works much better:

graphql server - this means easier management of queries/mutations, clear separation of data-service and UI code, and no flakey non-standard load/action stuff. "actions" are graphql mutations, and we don't have to specially form-encode hierarchical data to work with the action, the mutations just uses regular JSON, and it's all recorded in the schema for quick & easy error-checking.
tons of other tooling around graphql, in general (in-page data-inspection plugins, editor addons, query-generators, code-generators, etc)
It has full support for websockets, and they can be used easily, directly in graphql subscriptions
it runs on cloudflare worker, on miniflare locally.
I can access partially-remote service that uses remote DO + local KV, for a perfect combo of real data + fast/ephemeral dev (using cross-cf, a library I wrote)
we have full control of all of the stack, including how it is bundled
can work on the backend stuff separate from UI, even before it's built, with graphql-playground.
worker pages built by vite, which runs with a proxy to the local dev backend, so it all runs like 1 site - it has better hot-reload (which never worked right for me, with remix) and builds faster.

This isn't that different from how remix works (it also builds worker-pages, and locally proxies to miniflare), but we have control of everything, and it all works in a more standard & reliable way.

If remix works for you, that's great. It's probably something I am doing wrong, as a new-to-remix person, that makes it extra flakey and prone to breaking, but I am also an experienced react/edge-worker developer, and I just followed the directions on these projects sites (remix, mantine, etc) to set things up, so in my opinion, even if it's not a great failing of remix as a framework, the docs could use some work. Maybe everything that was breaking for me works again, but as I said, I will be slow to use or recommend it, in the future, considering the time I feel like I wasted with it.

kevinwolfcr commented 2 years ago

@konsumer I just followed this guide and it worked.

I am using all the latest versions from remix and mantine packages as of today.

konsumer commented 2 years ago

@fullstackjedi I'm glad it worked for you. The experience in Apr left a very bad taste in my mouth, around remix, and it will probably be a while before I try it on an actual project, again. My setup I replaced it with is still working very well, so I will stick with that.

shicholas commented 2 years ago

fwiw I can reproduce this with a new project setup and the latest versions of remix/react.

Viranchee commented 1 year ago

I have similar error, trying to troubleshoot, might drop remix

ErlanBelekov commented 1 year ago

still running into this!

grhbit commented 1 year ago

I use version 1.14.0 of remix. I got the same error. It seems to check whether it uses Node’s built-in module.

Here is my two cents.

Use an aliased name in package.json to avoid built-in module checking and run npm install or pnpm install (You might need to update npm if the npm version is too old. I'm using pnpm anyway.)
```
{
"dependencies": {
"buffer-polyfill": "npm:buffer@^6.0.3"
}
}
```

Create entry.client.ts and polyfill


import { RemixBrowser } from "@remix-run/react";
import { Buffer } from "buffer-polyfill";
import { hydrateRoot } from "react-dom/client";

globalThis.Buffer = Buffer as unknown as BufferConstructor;

hydrateRoot(document, );

lyquocnam commented 1 year ago

hope will have solution to resolve this problem

silence48 commented 1 year ago

globalThis.Buffer = Buffer as unknown as BufferConstructor;

This method works! using the latest remix and node 17.9.1

nastynaz commented 1 year ago

globalThis.Buffer = Buffer as unknown as BufferConstructor;
This method works! using the latest remix and node 17.9.1

What version of remix?

I'm still getting the issue with remix 1.15 and node 17.9.1

dabdine commented 1 year ago

Regarding the error above as it pertains to emotion/cloudflare/remix:

[mf:wrn] Parsed 1 valid header rule.
[mf:wrn] Service bindings are experimental. There may be breaking changes in the future.
[mf:err] ReferenceError: Buffer is not defined
    at node_modules/html-tokenize/node_modules/buffer-from/index.js (/private/Users/xxx/devel/corp-site/functions/[[path]].js:17750:59)
    at /private/Users/xxx/devel/corp-site/functions/[[path]].js:10:46
    at node_modules/html-tokenize/index.js (/private/Users/xxx/devel/corp-site/functions/[[path]].js:18569:22)
    at /private/Users/xxx/devel/corp-site/functions/[[path]].js:10:46
    at node_modules/@emotion/server/create-instance/dist/emotion-server-create-instance.cjs.dev.js (/private/Users/xxx/devel/corp-site/functions/[[path]].js:19886:49)
    at /private/Users/xxx/devel/corp-site/functions/[[path]].js:10:46
    at node_modules/@emotion/server/create-instance/dist/emotion-server-create-instance.cjs.js (/private/Users/xxx/devel/corp-site/functions/[[path]].js:19990:23)
    at /private/Users/xxx/devel/corp-site/functions/[[path]].js:10:46
    at /private/Users/xxx/devel/corp-site/functions/[[path]].js:45364:93
    at SourceTextModule.evaluate (node:internal/vm/module:226:23)

/private/Users/xxx/devel/corp-site/functions/[[path]].js:17750
    var toString5 = Object.prototype.toString, isModern = typeof Buffer.alloc == "function" && typeof Buffer.allocUnsafe == "function" && typeof Buffer.from == "function";

This worked for me on Remix 1.15.0 and react 18.2.0 using cloudflare pages: https://github.com/aiji42/remix-esbuild-override/tree/main/examples/emotion-cloudflare

Just follow what's in the README, with a slight tweak. Instead of:

const { withEsbuildOverride } = require("remix-esbuild-override");
const GlobalsPolyfills =
  require("@esbuild-plugins/node-globals-polyfill").default;

withEsbuildOverride((option, { isServer }) => {
  if (isServer)
    option.plugins = [
      GlobalsPolyfills({
        buffer: true,
      }),
      ...option.plugins,
    ];

  return option;
});

Use this (just prepend to your remix.config.js):

const { withEsbuildOverride } = require("remix-esbuild-override");

withEsbuildOverride((option, { isServer }) => {
  if (isServer) {
    option.inject = [
      ...(option.inject ?? []),
      require.resolve("@esbuild-plugins/node-globals-polyfill/Buffer.js"),
      require.resolve("@esbuild-plugins/node-globals-polyfill/process.js"),
    ];
  }

  return option;
});

Which came from this issue: https://github.com/aiji42/remix-esbuild-override/issues/35#issuecomment-1385058514

huw commented 1 year ago

This has (arguably) been fixed upstream by Cloudflare—you should be able to opt-in to polyfilling Buffer and other Node globals if you want.

jdnichollsc commented 1 year ago

Hey folks, check my example to fix this issue https://gist.github.com/jdnichollsc/b73eca1f34bdd1e3a95283774fda2212

VishakhaRastogi commented 1 year ago

I am stilling facing this issue even after making changes in remix.config.js told in this comment https://github.com/remix-run/remix/issues/2813#issuecomment-1528928962

Uncaught ReferenceError: Buffer is not defined
    at node_modules/@redis/client/dist/lib/client/commands-queue.js (commands-queue.js:299:18)
    at __require2 (chunk-PZDJHGND.js:18:50)
    at node_modules/@redis/client/dist/lib/client/index.js (index.js:17:26)
    at __require2 (chunk-PZDJHGND.js:18:50)
    at node_modules/@redis/client/dist/index.js (index.js:18:18)
    at __require2 (chunk-PZDJHGND.js:18:50)
    at node_modules/redis/dist/index.js (index.js:18:18)
    at __require2 (chunk-PZDJHGND.js:18:50)
    at redis.tsx:1:50

Using remix version 1.18.1

markdalgleish commented 1 year ago

The buffer polyfill is built into Remix when targeting non-Node server platforms, so similarly to https://github.com/remix-run/remix/issues/2813#issuecomment-1458138043, you should be able to import the built-in buffer polyfill and assign it to globalThis.Buffer at the root of your app without needing to install any additional packages.

Note that in the upcoming Remix v2 you'll need to explicitly enable this polyfill via the serverNodeBuiltinsPolyfill option.

markdalgleish commented 1 year ago

@jtan80813 Have you manually put Buffer on global?

import { Buffer } from "buffer";

globalThis.Buffer = Buffer;

jtan80813 commented 1 year ago

@markdalgleish. Yes I did in entry.client.ts, right?

markdalgleish commented 1 year ago

If you're using Cloudflare Workers, you might need this global polyfill on the server too.

flatoy commented 11 months ago

https://developers.cloudflare.com/workers/configuration/compatibility-dates/#nodejs-compatibility-flag

have you tried this?

tleperou commented 11 months ago

https://developers.cloudflare.com/workers/configuration/compatibility-dates/#nodejs-compatibility-flag

have you tried this?

Just tried; is not working for me.

But you can you it that way https://developers.cloudflare.com/workers/runtime-apis/nodejs/buffer/ – if you access the code running it though.

remix-run / remix

Buffer is not defined #2813

What version of Remix are you using?

Steps to Reproduce

Expected Behavior

Actual Behavior