How to use esbuild with Cloudflare Workers?

[rfgamaral]

I first created a Cloudflare Worker project with:

wrangler generate my-graphql-server https://github.com/signalnerve/workers-graphql-server

And then ran npm install followed by wrangler dev. A server was available at http://127.0.0.1:8787, and the GraphQL server seemed to be working (despite having a bunch of errors related to chokidar). The next step was to replace the default Webpack bundler with esbuild. I have a demo repository available here. To test esbuild:

• Clone the repository
• Rename wrangler.toml to wrangler.webpack.toml
• Rename wrangler.esbuild.toml to wrangler.toml
• Run wrangler dev

*You'll need to install wrangler as a global dependency, have a Cloudflare account and change the account_id in the `.toml` file to actually test this through.**

The above setup is not working with esbuild... After running wrangler dev, a server is available at http://127.0.0.1:8787. Opening that link on a browser results in a web page with a "Worker threw exception" error page, and the following error on the console:

Uncaught
ReferenceError: require is not defined
    at worker.js:1:16
Uncaught (in response)
ReferenceError: require is not defined

I'm not quite sure what I'm missing here, and I'd love to use esbuild with Cloudflare Workers, if possible. Hopefully this is a bad configuration on my side, and not an incompatibility on esbuild or Cloudflare Workers.

Any help is appreciated 🙏

[lukeed]

It's possible. I have a cfw package which is just an esbuild wrapper. All my workers are built with it.

It works with most projects ootb. Important things to note are the target version and output format.

[rfgamaral]

@lukeed Can you share the optimal target and output options for Cloudflare Workers? I been tried a few combinations, but I can't get the project above working.

[lukeed]

@rfgamaral https://github.com/lukeed/cfw/blob/master/src/commands/build.ts#L9-L21

My target is usually esnext, only cuz I don't use much modern syntax. To be safe, you probably want your target (either directly or via tsconfig) to be es2019

[rfgamaral]

@lukeed Tried a few combinations taking into account your code, but I can't get it working, same issue, require is not defined 😥

[evanw]

ReferenceError: require is not defined

One cause of this could be that you are using a package which uses require in some dynamic form that esbuild can't bundle such as require(someVariable). There used to be a warning that may have triggered when this happens, but many people complained about the warning so it has been removed, which makes these issues harder to debug. Can you describe what the code that ends up calling require looks like, and where it comes from?

[rfgamaral]

@evanw The source can be seen here, and I'm building with the esbuild ./src/index.js --platform=node --bundle --outfile=./dist/index.js command. And here's a gist with the output.

[rfgamaral]

I've researched a bit more and found this Cloudflare blog post:

• Node.js support in Cloudflare Workers

Somewhere in there:

For over 20k packages, Workers supports this magic already: any Node.js package that uses webpack or another polyfill bundler runs within our environment today. You can get started with the greatest hits packages like node-jose for encryption, itty-router for routing, graphql for querying your API, and so much more.

In other words, I don't think I'm meant to build with --platform=node, however, when I build with esbuild ./src/index.js --bundle --outfile=./dist/index.js, I get the following errors:

> esbuild ./src/index.js --bundle --outfile=./dist/index.js

 > node_modules/graphql-upload/lib/processRequest.js:6:43: error: Could not resolve "util" (use "--platform=node" when building for node)
    6 │ var _util = _interopRequireDefault(require('util'))
      ╵                                            ~~~~~~

 > node_modules/busboy/lib/main.js:1:17: error: Could not resolve "fs" (use "--platform=node" when building for node)
    1 │ var fs = require('fs'),
      ╵                  ~~~~

 > node_modules/busboy/lib/main.js:2:29: error: Could not resolve "stream" (use "--platform=node" when building for node)
    2 │     WritableStream = require('stream').Writable,
      ╵                              ~~~~~~~~

 > node_modules/busboy/lib/main.js:3:23: error: Could not resolve "util" (use "--platform=node" when building for node)
    3 │     inherits = require('util').inherits;
      ╵                        ~~~~~~

 > node_modules/apollo-server-cloudflare/node_modules/apollo-server-core/dist/utils/createSHA.js:9:30: error: Could not resolve "crypto" (use "--platform=node" when building for node)
    9 │         return module.require('crypto').createHash(kind);
      ╵                               ~~~~~~~~

 > node_modules/fs-capacitor/lib/index.js:8:45: error: Could not resolve "crypto" (use "--platform=node" when building for node)
    8 │ var _crypto = _interopRequireDefault(require("crypto"));
      ╵                                              ~~~~~~~~

 > node_modules/apollo-server-cloudflare/node_modules/apollo-engine-reporting/dist/agent.js:16:37: error: Could not resolve "os" (use "--platform=node" when building for node)
    16 │ const os_1 = __importDefault(require("os"));
       ╵                                      ~~~~

 > node_modules/apollo-server-cloudflare/node_modules/apollo-engine-reporting/dist/agent.js:17:23: error: Could not resolve "zlib" (use "--platform=node" when building for node)
    17 │ const zlib_1 = require("zlib");
       ╵                        ~~~~~~

 > node_modules/fs-capacitor/lib/index.js:10:41: error: Could not resolve "fs" (use "--platform=node" when building for node)
    10 │ var _fs = _interopRequireDefault(require("fs"));
       ╵                                          ~~~~

 > node_modules/apollo-server-cloudflare/node_modules/apollo-env/lib/fetch/url.js:3:20: error: Could not resolve "url" (use "--platform=node" when building for node)
    3 │ var url_1 = require("url");
      ╵                     ~~~~~

10 of 25 errors shown (disable the message limit with --log-limit=0)

The same doesn't happen when using Webpack, and everything works with Cloudflare Workers when using Webpack. Maybe Webpack is bundling util, stream, os, crypto, and others in the output? Here's a gist with the output from Webpack.

Assuming that's the problem, is it possible to configure esbuild to also bundle those libraries just like Webpack?

[evanw]

@evanw The source can be seen here, and I'm building with the esbuild ./src/index.js --platform=node --bundle --outfile=./dist/index.js command. And here's a gist with the output.

The error is likely thrown by the require('util') code in the output file. The problem is that you are running esbuild with --platform=node but then not running the output file in node. The JavaScript environment in Cloudflare Workers is not node, so that makes sense why an output file meant for node would crash in Cloudflare Workers.

Although esbuild isn't designed with Cloudflare Workers in mind at all, you may be able to get this to work by setting --platform=neutral to disable all platform-specific behavior (since you are not running your code in either the browser or in node, the only two platforms esbuild supports). See https://esbuild.github.io/api/#platform for details. But you will be on your own, and will have to deal with all platform-specific nuances yourself. For example, you may not be able to use libraries designed for node, since you're not running the code in node. If that's too much trouble, then feel free to use Webpack instead of esbuild.

[rfgamaral]

Tried with --platform=neutral but it didn't work either, got this instead:

> esbuild ./src/index.js --bundle --platform=neutral --outfile=./dist/index.js

 > src/handlers/apollo.js:1:29: error: Could not resolve "apollo-server-cloudflare" (mark it as external to exclude it from the bundle)
    1 │ import { ApolloServer } from 'apollo-server-cloudflare';
      ╵                              ~~~~~~~~~~~~~~~~~~~~~~~~~~

 > src/schema.js:1:20: error: Could not resolve "apollo-server-cloudflare" (mark it as external to exclude it from the bundle)
    1 │ import { gql } from 'apollo-server-cloudflare';
      ╵                     ~~~~~~~~~~~~~~~~~~~~~~~~~~

 > src/datasources/pokeapi.js:1:31: error: Could not resolve "apollo-datasource-rest" (mark it as external to exclude it from the bundle)
    1 │ import { RESTDataSource } from 'apollo-datasource-rest';
      ╵                                ~~~~~~~~~~~~~~~~~~~~~~~~

 > node_modules/apollo-server-cloudflare/dist/cloudflareApollo.js:13:37: error: Could not resolve "apollo-server-core" (mark it as external to exclude it from the bundle, or surround it with try/catch to handle the failure at run-time)
    13 │ const apollo_server_core_1 = require("apollo-server-core");
       ╵                                      ~~~~~~~~~~~~~~~~~~~~

 > node_modules/apollo-server-cloudflare/dist/cloudflareApollo.js:14:36: error: Could not resolve "apollo-server-env" (mark it as external to exclude it from the bundle, or surround it with try/catch to handle the failure at run-time)
    14 │ const apollo_server_env_1 = require("apollo-server-env");
       ╵                                     ~~~~~~~~~~~~~~~~~~~

5 errors

If that's too much trouble, then feel free to use Webpack instead of esbuild.

I didn't really want to use Webpack, esbuild looks so much better and faster, but I guess I don't have a choice... 😞

[evanw]

If you read the docs for platform linked above, it mentions that the neutral platform disables all of esbuild's platform-specific defaults including the main fields setting. Setting --main-fields=main will help get most node-style packages to build.

For over 20k packages, Workers supports this magic already: any Node.js package that uses webpack or another polyfill bundler runs within our environment today. You can get started with the greatest hits packages like node-jose for encryption, itty-router for routing, graphql for querying your API, and so much more.

You can try this approach with esbuild as well. In that case you could call esbuild's JS API, use platform: 'node' like you were doing originally, and write a small plugin that redirects imports to node's built-in modules such as util with your choice of polyfill modules. The plugin API you'd want to use is here: https://esbuild.github.io/plugins/#resolve-callbacks. Although Webpack 4 automatically polyfills node APIs with browser-specific versions by default, esbuild doesn't (and I believe Webpack 5 also doesn't) so you will have to bring your own polyfills.

[rfgamaral]

@evanw Thank you, I think I understand this now.

I wasn't able to get the example project I posted initially working, but I was close (it's missing the auto polyfill bit, but I can't be bothered by that now).

However, I was able to apply what I learned here into my real project, and that one built successfully. Thank you so much.

[brillout]

If you read the docs for platform linked above, it mentions that the neutral platform disables all of esbuild's platform-specific defaults

Has anyone achieved Cloudflare Workers blunding with platform: "neutral"?

Many vite-plugin-ssr users use esbuild with platform: "browser" which is problematic in many ways.

@threepointone @petebacondarwin what do you use for wrangler 2?

@evanw How about something like platform: 'cloudflare-workers' or platform: 'worker'?

CC @dan-lee @Aslemammad

[petebacondarwin]

Hi @brillout. You can see who we configure esbuild in Wrangler2 for compiling a Worker for upload here: https://github.com/cloudflare/wrangler2/blob/85392b33d0a9e3eef7e4089dd97a9428b5fea730/packages/wrangler/src/bundle.ts. Also, the "pages" system that sits inside Wrangler uses esbuild too: https://github.com/cloudflare/wrangler2/blob/85392b33d0a9e3eef7e4089dd97a9428b5fea730/packages/wrangler/pages/functions/buildWorker.ts#L14

Many vite-plugin-ssr users use esbuild with platform: "browser" which is problematic in many ways.

Perhaps you could elaborate on the problems with this?

[BasixKOR]

Webpack provides a webworker target which Cloudflare currently recommends for and resolves to worker exports field if found. We could take inspiration from them.

[brillout]

Perhaps you could elaborate on the problems with this?

The main problem is that esbuild resolves the wrong value of package.json#exports. But I'm just seeing the conditions config; that should do the trick.

[brillout]

I've released https://github.com/brillout/build-worker. It's a thin wrapper that configures esbuild for Cloudflare Workers.

This esbuild config: https://github.com/brillout/build-worker/blob/master/build-worker.mjs.

This will be the official recommendation for vite-plugin-ssr 0.4. (FYI vite-plugin-ssr.com.)

CC @dan-lee.