Support the esbuild plug-in system？

[ycjcl868]

esbuild is small but complete in every detail, do you have any plans to support the plugin-in system to extend the web development workflows?

[Jarred-Sumner]

@lukeed if you want I have a fork where I manually published the branch with plugins, just add this to your dependencies inside package.json:

"esbuild": "https://github.com/Jarred-Sumner/esbuild/releases/download/pluginbuild/esbuild-0.8.1.tgz",

The only change I made was modifying the install.ts script to not download from NPM/local cache and instead fetch from that github release page (plus version bump so I could be sure it wasn't the non-plugins version)

[evanw]

Getting the initial version of the plugin API out is going to be the focus of my next big release.

One thing I'm worried about is the performance of the JavaScript API. Right now I have primarily optimized the JavaScript API for ease of use. A plugin is just a JavaScript function that can make a few different calls to the plugin API functions. This makes plugins very lightweight because they are small and easy to write inline in the same file:

let examplePlugin = plugin => {
  plugin.setName('example')
  plugin.addLoader({ filter: /.*/ }, async (args) => {
    return { content: await require('fs').promises.readFile(args.path, 'utf8') }
  })
}

await esbuild.build({
  plugins: [examplePlugin],
  ...
})

However, that comes at the expense of some performance opportunity cost because JavaScript is single-threaded. This can potentially be very slow if you are running every input file through a plugin because the build is bottlenecked through a single CPU. There's an argument to be made that the plugin API should focus on ease of use, and also that JavaScript plugins are doomed to be slow anyway. But given the focus of esbuild on pushing for faster tools I think performance should be a strong consideration, and this is arguably especially important for JavaScript-based plugins given the performance handicap.

So I think it's important to investigate the performance impact of a few different API designs for the plugin API during the design phase. Obviously the plugin API can still be changed after releasing it but I think the investigation won't take that long and doing it ahead of time avoids releasing an API only to potentially change it immediately. Sorry about the delay. I know the plugin API is really exciting but I think it's prudent to be careful and deliberate about these design decisions.

My current idea for improving performance is to change the plugin API such that plugins must be in separate files. You would then pass a file name to esbuild and esbuild would spin up several node child processes that handle plugin invocations completely in parallel. Maybe something like this:

// example-plugin.js
module.exports = options => ({
  name: 'example',
  setup(build) {
    plugin.onLoad({ filter: /.*/ }, async (args) => {
      return { content: await require('fs').promises.readFile(args.path, 'utf8') }
    })
  },
})

// build.js
await esbuild.build({
  plugins: {
    './example-plugin.js': { /* options */ },
  },
  ...
})

Plugins that contain synchronized global data structures will be unable to run in parallel. This should be easy to handle by just running them in the host node process instead of in the child node processes. Perhaps they could be specified in a separate serialPlugins property instead of mixing them in with plugins.

I'm going to investigate this approach next and report back with performance numbers.

[intrnl]

I think converting the plugin function to string would work the same way and should avoid the need for esbuild to resolve files on its own, though that could mean the function needs to be async for it to work with ESM.

function sveltePlugin () {
  let { compile } = require('svelte');

  return {
    name: 'svelte',
    // ...
  };
}

[evanw]

I think converting the plugin function to string would work the same way and should avoid the need for esbuild to resolve files on its own

That would break relative imports:

function sveltePlugin () {
  let { compile } = require('./svelte.js');

  return {
    name: 'svelte',
    // ...
  };
}

The code would no longer be able to resolve ./svelte.js when run because esbuild wouldn't know what directory to run it in, since all it has is a function object.

I immediately hit the problem of esbuild needing to resolve files itself when I tried implementing this, so you're totally right that the specific API shape I proposed isn't good. Right now I'm just requiring the caller to run require.resolve() first while I focus on the proof-of-concept demo. This API shape will need more thought if the plugin API ends up going in this direction.

[evanw]

Here's an update. Sorry about the long post.

I'm testing plugin parallelism by running each file in my JavaScript benchmark through a plugin that runs the TypeScript compiler's transpileModule function. This is a no-op because the input files are already JavaScript, but it should be a realistic performance test. After all, there is no difference in speed between esbuild's JavaScript and TypeScript parsers.

The results are disappointing. I was hoping I could get at least a 2x speedup for JavaScript plugins using parallelism because esbuild can do that easily, but I the maximum speedup was 1.5x. I was also hoping that having esbuild create the child processes would be more efficient than having the plugin do that itself because it would mean using completely parallel IPC channels instead of having everything being bottlenecked through the single IPC channel with the host process, but it turns out the IPC channel is so fast that the numbers basically don't matter given how slow the TypeScript compiler is.

Here's the data from my tests:

Count	GOMAXPROCS	Child process of plugin	Child process of esbuild
1	1.64s	18.55s	19.08s
2	0.92s	12.53s	13.12s
3	0.69s	12.00s	12.01s
4	0.57s	12.20s	13.19s

Each time is the best of three runs and there is probably still a little bit of noise. Here's what each column means:

1. Count is the number of units of parallelism, where 1 means no parallelism (i.e. 1 CPU).
2. "GOMAXPROCS" is a baseline measurement of esbuild doing the TypeScript-to-JavaScript conversion without using any plugins (GOMAXPROCS controls the number of OS threads used by Go).
3. "Child process of plugin" means the plugin itself uses node's child_process.fork() function to create long-lived child node processes that run transpileModule().
4. "Child process of esbuild" means that esbuild uses Go's exec.Command() function to create long-lived child node processes that run transpileModule().

Here is that same data divided by the first row in each column. This represents the relative speedup obtained by using parallelism:

Count	GOMAXPROCS	Child process of plugin	Child process of esbuild
1	1.00x	1.00x	1.00x
2	1.78x	1.48x	1.44x
3	2.38x	1.55x	1.51x
4	2.88x	1.52x	1.37x

And here is the relative speedup visualized:

Note that it's not possible for these speedup multiples to be exactly equal to the count because some of aspects of bundling cannot be parallelized. Also at some point increasing the count will actually decrease performance because there is a limited number of cores and using more parallel tasks than the number of cores results in wasted effort switching between tasks. My machine has 6 cores so theoretically times should continue to improve up to a count of 6.

While esbuild continues to improve by a significant amount for each unit of parallelism, the TypeScript plugin essentially only benefits from having one additional child process. After that point the results barely improve and then start dropping. This was unintuitive for me and is very unfortunate because it means JavaScript plugins in esbuild are pretty much guaranteed to be slow, especially as compared to Go plugins.

After some investigation I believe I have an explanation: although JavaScript the language is single-threaded, V8 is not and usually uses 200-300% of your CPU (as confirmed by top). I know that at least V8's GC is parallelized, and various aspects of the JIT compiler may be too. I tried to find flags to force V8's GC to run on the main thread but nothing I tried seemed to have an effect.

I think the conclusion is that parallelism of JavaScript plugins isn't a big win and that it probably doesn't make sense to parallelize most plugins. It probably only makes sense to parallelize your primary plugin and even then it's probably not worth creating more than two child processes. Given that parallelism of JavaScript plugins isn't going to be a big focus, right now I'm thinking that it's probably ok for the plugin to just do that itself using child_process instead of building plugin parallelism into esbuild. That means I could move forward with releasing the current plugin API without any fundamental changes.

[Jarred-Sumner]

After some investigation I believe I have an explanation: although JavaScript the language is single-threaded, V8 is not and usually uses 200-300% of your CPU (as confirmed by top). I know that at least V8's GC is parallelized, and various aspects of the JIT compiler may be too. I tried to find flags to force V8's GC to run on the main thread but nothing I tried seemed to have an effect.

This is probably a really dumb idea, but what if you tried using one of the slower, embeddable JavaScript interpretors (without a JIT) instead of V8 for plugins? For example: https://bellard.org/quickjs/ or maybe https://github.com/facebook/hermes. https://github.com/robertkrimen/otto might be easiest to try since its already written in Go.

[evanw]

what if you tried using one of the slower, embeddable JavaScript interpretors (without a JIT) instead of V8 for plugins?

I have some experience with this. While I haven't tried this with esbuild specifically, we did do a performance investigation when we switched to QuickJS for Figma plugins. V8 with a JIT is an order of magnitude faster than non-JIT interpreters which makes switching away from V8 for esbuild a non-starter. You can see some benchmarks here: https://bellard.org/quickjs/bench.html.

There is also the issue of the API exposed to plugins. The point of plugins in esbuild is to integrate with other packages from node's ecosystem, and they expect to use node's full API including all built-in libraries and support for native binary extensions. Replicating all of that on top of another JavaScript interpreter would be a very large (and ongoing) amount of work, so that is also a non-starter.

[Jarred-Sumner]

That makes sense

Another thought, which might be a bad idea: what if a very limited subset of Node’s internals are replaced by an RPC for the esbuild process. Eg override fs.readFile and fs.writeFile to be implemented in the Go process instead of Node.

Another way of asking this question is, what if it’s Node.js internals that’re worse at parallelism than V8?

On Sat, Oct 31, 2020 at 3:40 AM Evan Wallace notifications@github.com wrote:

what if you tried using one of the slower, embeddable JavaScript interpretors (without a JIT) instead of V8 for plugins?

I have some experience with this. While I haven't tried this with esbuild specifically, we did do a performance investigation when we switched to QuickJS for Figma plugins. V8 with a JIT is an order of magnitude faster than non-JIT interpreters which makes switching away from V8 for esbuild a non-starter. You can see some benchmarks here: https://bellard.org/quickjs/bench.html.

There is also the issue of the API exposed to plugins. The point of plugins in esbuild is to integrate with other packages from node's ecosystem, and they expect to use node's full API including all built-in libraries and support for native binary extensions. Replicating all of that on top of another JavaScript interpreter would be a very large (and ongoing) amount of work, so that is also a non-starter.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/evanw/esbuild/issues/111#issuecomment-719916079, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFNGS77YYR3VVHOVMNM3NLSNPSTTANCNFSM4NAM6XYA .

[hronro]

Does the plugins have to be based on JavaScript? For me, if I need plugins written in JavaScript and I don't care about performance, I would use Babel. After all, babel has a larger ecosystem than esbuild for now. But I do care about performance. So if JavaScript based plugins would slow down esbuild, I would perfer writing plugins in some native languages and then communicate with esbuild via MessagePack for example.

[bep]

Does the plugins have to be based on JavaScript?

There will be a similar Go API (I'm looking forward to using that in Hugo), but that would not allow you to interact with JS libraries, e.g. the Svelte compiler.

[airhorns]

Not to pile on, but as a user trying to get away from Babel because it's too slow, I'd love it if the esbuild plugin story had a far higher performance ceiling than Babel's. I agree with @foisonocean that that's something special that esbuild could do: offering users a novel point along the investment/performance tradeoff curve. It makes sense that most plugin authors would want to use the same language they're bundling to write plugins, but, esbuild kind of proves that you are destined for a slow build process if you do that, so I would love the option to invest to go fast.

I suggest allowing WASM plugins instead of JS. This would allow plugins to run in a variety of languages including high performance ones, and they could be run server side using a fancy fast golang WASM VM and then potentially natively by the browser when using the WASM compiled version of esbuild there. You don't have to build or worry about an IPC channel, you don't have to deal with child processes, and you can parallelize using Go's plain old concurrency primitives. The WASM spec (mostly) handles the interface definitions such that you wouldn't have to define and evolve a protobuf or messagepack schema or whatever for IPC which is nice too.

It'd be awesome to just go write a little bit of AssemblyScript or golang to implement tiny replacements for the last couple babel plugins I have to run and fully realize the 100x bundling speedup.

[Ventajou]

I went from a Webpack build to a 2 stage setup where esbuild runs first and then Webpack processes the result. I have a handful of tasks that esbuild just can't do without plugins. It's a bit of a complex setup but it's already significantly faster than the original.

Even a single-threaded JS plugin system would be an improvement for me both in speed and complexity. And it would be relatively easy to adopt since plugins are often just glue for other Node libraries. I think it would allow a lot of people to start using esbuild and achieve much faster build times. And the plugin ecosystem would grow really quickly.

If we also have the possibility to write plugins in Go (and use both Go and JS plugins in the same build), it gives people a chance to target the slowest plugins and rebuild them in Go. I'm sure people will upgrade to faster plugins as they become available.

esbuild audience is JS developers and only a subset of them will have the knowledge or time to invest into writing plugins in another language. And there will always be the odd task that's very specific to a team and they'll have to write a plugin for it, JS will be the quick and efficient answer for that.

[lukeed]

@evanw Thanks for the insights – unfortunate, but not super surprising. Spawning processes is expensive. I wonder if it might make a difference if using worker_threads instead, as opposed to a full process. It'd still require separate plugin files, but since the plugins are known upfront (and their options, presumably), it should still be possible to spawn and track a pool of threads.

(I haven't really done anything in this space, so I can't really help much aside from doc links)

[leeoniya]

I wonder if it might make a difference if using worker_threads instead, as opposed to a full process.

if this strategy allows for keeping the JIT'd code cached rather than being thrown away and re-JIT'd on each process spawn, it could make a significant difference.

[lukeed]

Right, they're significantly cheaper to create, but I believe there's a limit to how many can be spawned.

[evanw]

Does the plugins have to be based on JavaScript?

They don't have to, no. The plugin API is available in both JavaScript and Go. My intuition is that the vast majority of libraries people want to use with esbuild are written in JavaScript, so it seems like a mistake to not have a JavaScript plugin API.

But I do care about performance. So if JavaScript based plugins would slow down esbuild, I would perfer writing plugins in some native languages and then communicate with esbuild via MessagePack for example.

This is possible with the Go API. You can create a Go project which calls esbuild's API with a plugin that communicates with code written in other languages however you want (pipes, child processes, network calls). There wouldn't be any JavaScript executed at all in that scenario.

I suggest allowing WASM plugins instead of JS.

My thought was that since Go is native code and pretty much anything is possible, it should be possible for people to start experimenting with things like this using esbuild's Go API without having to build a whole WASM VM into esbuild itself. There shouldn't be a performance penalty for doing this in a plugin instead of doing this within esbuild.

I wonder if it might make a difference if using worker_threads instead, as opposed to a full process.

I just tried this and the performance numbers are the same as the other two child process approaches.

[leeoniya]

I just tried this and the performance numbers are the same as the other two child process approaches.

that seems odd, is the case here of spawning a process vs a thread and then terminating them and re-spawning later? (https://stackoverflow.com/a/60488780/973988).

if each plugin is a persistent pure/functional worker thread and you're simply feeding stuff to it via .postMessage, the overhead should be on the order of 10ms in extreme cases (certainly an order of magnitude faster than invoking the TS compiler)?

[1] https://wanago.io/2019/05/13/node-js-typescript-13-sending-data-worker-threads/ [2] https://www.jefftk.com/p/overhead-of-messagechannel

[evanw]

that seems odd, is the case here of spawning a process vs a thread and then terminating them and re-spawning later? (https://stackoverflow.com/a/60488780/973988).

In all scenarios, each plugin VM is created once at the start of the build and is then persistent throughout the entire build.

if each plugin is a persistent pure/functional worker thread and you're simply feeding stuff to it via .postMessage, the overhead should be on the order of 10ms (certainly an order of magnitude faster than invoking the TS compiler)?

In all scenarios, the overhead of message passing was pretty insignificant relative to the overall build time. It's possible to have high throughput even with higher latency because JavaScript is slower than Go (so it's the bottleneck) and Go keeps JavaScript's input queue full throughout the build.

[hronro]

This is possible with the Go API. You can create a Go project which calls esbuild's API with a plugin that communicates with code written in other languages however you want (pipes, child processes, network calls). There wouldn't be any JavaScript executed at all in that scenario.

From my experience, most users of babel/esbuid don't write plugins, all they do are downloading babel/esbuild and plugins from NPM and writing a configuration file for babel/esbuild, they don't care about which programming language are used in esbuild and its plugins. The current Go API requires users to create a Go project, which is really hard for most users. So I think it would be better if the "Go plugins" can also be used with a simple npm install command and one additional line in the configuration.

More futher, I want to talk about NeoVim's plugin system. NeoVim allows users to write plugin in any languages by using MessagePack. MessagePack is a binary serialization format used for RPC. I haven't look deep into the RPC call overheads, but MessagePack claims it's very efficient and from my experience it's faster than protobuf. Imagine I could write plugins in any language, which means I can write plugins in Go, I can also write plugins in some more performant languesges like c/c++/rust, and I can also write plugins in JavaScript if I have to interact with JS libraries like the Svelte compiler. Yes many RPC solutions also has NodeJS bindings, so esbuild just need to mantain one generic RPC API instead of mantaining two separate JS/Go APIs.

[evanw]

I just released the initial version of the plugin API. It's somewhat different than the version on the plugins branch:

• Plugins are now objects with a name property and a callback instead of just being a function. I made this change because it lets you get the plugin name without invoking the plugin, which could potentially be important in some future scenarios.

• I also renamed "addResolver" and "addLoader" to "onResolve" and "onLoad" to distinguish between the similarly-named built-in concepts, and to make it natural to add more hooks in the future whose names don't lend themselves to verbs.

There is an example of this new format in the release notes. The new plugin API isn't documented yet though. I'm still working on the documentation and I plan to land it sometime in the next few days.

[evanw]

So I think it would be better if the "Go plugins" can also be used with a simple npm install command and one additional line in the configuration.

Yes, point heard. I think the approach you're proposing is interesting. However, I think it can coexist in parallel with the existing API instead of replacing it. I agree that there is a certain elegance to having everything use RPC but there are some drawbacks around ease of use and performance/memory usage. I think the existing API is good to have as an easy entry point and I RPC-based plugins could be left as a more advanced use case.

Right now a JavaScript plugin looks like this:

{ name: 'example', setup(build) { ... } }

With an RPC system, a plugin could look like this instead assuming there's a binary executable called binary in the same directory:

{ name: 'example', executable: path.join(__dirname, 'binary') }

Then esbuild would launch that executable as a child process and communicate with it over stdin/stdout using some RPC protocol. From the user's perspective they wouldn't even know the difference because in both scenarios they would just require('example-esbuild-plugin') which would return one of these objects.

There could also be support for network-based plugins using the same RPC protocol over a TCP stream:

{ name: 'example', network: 'localhost:8080' }

This could help in scenarios where you are doing many builds in quick succession and you want to use a plugin written in an inefficient language with a long start-up time (e.g. Java), so you need to put the plugin in a long-lived process for performance.

You wouldn't want to use JavaScript with either of these options for the reasons described above in this thread. Each node instance uses multiple CPU cores and you'll quickly run out of CPUs if you do that. Instead, JavaScript plugins should be co-located in the host process using esbuild's normal JavaScript plugin API.

One thing I also recently added is the ability for plugins to proxy other plugins. Basically you can now return a different plugin name from a plugin callback if you're proxying for that plugin. That should allow for a single executable or network connection in the above proposal to potentially represent multiple plugins if that's ideal from a performance/memory perspective for your use case.

An interesting thing about this proposal is that it would also make it possible to add the ability to run certain plugins from the CLI directly. I'm imagining flags like --plugin-exec:./path/to/executable and --plugin-net:localhost:8000.

I'm not planning on implementing this proposal immediately because I want to get the initial plugin API to a good place and then fix some other pressing issues. But I do think this direction could be useful for more advanced cases. I'm sure people will want to use it to write plugins in Rust, for example :)

[eigilsagafos]

@evanw Would it make sense to expose the esbuild configuration options to the plugins? I would like to be able to read both external (or have externals ignored by the plugin) and platform to mark node builtins as external.

[evanw]

FYI Plugin documentation is up now: https://esbuild.github.io/plugins/

@evanw Would it make sense to expose the esbuild configuration options to the plugins? I would like to be able to read both external (or have externals ignored by the plugin) and platform to mark node builtins as external.

Yes. That's come up before and is something I'm planning on doing. Right now I'm thinking that the options object could just always be provided to the plugin inside the setup function.

[chowey]

Is there a recommended way to distribute Go plugins? I can't think of anything clever. It seems like I need to re-compile esbuild with my plugin.

[evanw]

Is there a recommended way to distribute Go plugins? I can't think of anything clever. It seems like I need to re-compile esbuild with my plugin.

That's correct. The Go API is intended to be used by Go projects. Go plugins are intended to be distributed the normal way you distribute Go code (e.g. publishing on GitHub is sufficient).

Are you are trying to publish a package written in a native language to npm so it can be called by people who are using esbuild from JavaScript? It's possible to do this but it's a lot of work:

• You'd have to use node's child_process module to run the native code and you'd have to implement communication with it yourself (e.g. RPC over stdin/stdout like esbuild).

• You would have to figure out how to build and publish your native executable for all of the different architectures you want to support (esbuild supports 9 so far, for example).

• If you don't want installing your package to download the binaries for all architectures, you'll need an install script that detects the architecture and downloads the correct one, and you'll need to find some method to host them (I use separate packages on npm for esbuild).

  Getting an install script to work for everyone is tricky because people have lots of custom needs that need special handling (e.g. proxies, alternatives to npm, alternative package managers). It took me a while to get esbuild's install script to a good place.

[chowey]

Mainly I wanted to be able to write a Go plugin and have it usable by other members of the team. I understand now the use case for a Go plugin:

The Go API is intended to be used by Go projects.

This is totally fine.

It seems obvious now that if I want to create a plugin for the NPM ecosystem then I would then have to deal with the NPM ecosystem. It wasn't really what I wanted to do anyway. Go plugins for Go projects will work fine for me.

[zmitry]

@evanw I want to congratulate you with great progress on watch mode, it's really impressive that you've got there so fast. I was curious is it possible to implement something like WebpackHtmlPlugin with current plugin api? Or do you have plans to support that natively? Html support was last blocker for me to adopt esbuild, will you accept some help on that? So from what I can see plugins api is not enough to support this feature and html as first class citizen requires some changes in the core and I'm not sure it's in your roadmap.

[evanw]

FYI: I just created a repo for issue #632 to serve as a centralized list of 3rd-party esbuild plugins:

https://github.com/esbuild/community-plugins

I didn't want to pre-populate the list with existing esbuild plugins because I didn't want to make any assumptions for plugin authors that they want their plugins on the list. So the list is currently empty. Feel free to add your plugin to that list if you think it should be there!

[luncheon]

Currently, we need to connect onResolve and onLoad in namespace. In some use cases, the namespace is overkill and a bit troublesome.

let envPlugin = {
  name: 'env',
  setup(build) {
    build.onResolve({ filter: /^env$/ }, args => ({
      path: args.path,
      namespace: 'env-ns',
    }))

    build.onLoad({ filter: /.*/, namespace: 'env-ns' }, () => ({
      contents: JSON.stringify(process.env),
      loader: 'json',
    }))
  },
}

It would be nice if the above example could be written as below.

let envPlugin = {
  name: 'env',
  setup(build) {
    build.onResolve({ filter: /^env$/ }, () => ({
      contents: JSON.stringify(process.env),
      loader: 'json',
    }))
  },
}

Is there a problem with onResolve being able to return the contents directly?

[myagoo]

Hi there, I did not real all of the above, so sorry if the question has already been asked or is completely dumb. Can't namespaces be inferred from the name of the loader ? I expect loader name to be unique (from the package name to the name property of the plugin within a bundler instance). This way, namespaces conflict would'nt be a thing. Or maybe you wanted for different plugins to be able to share the same namespace ?

Anyway, amazing tech and documentation work :clap:

[hardfist]

@evanw It would be very convenient if load can return sourcemap with contents(even thou user can embed sourcemap into content themselves)

[RIP21]

@evanw hey. Thanks for an amazing project. I have read whole thread but still have a question.

Am I right that current plug-ins system is suitable for certain extensions, but not for modifying AST. E.g. if there is a need to change let say, lodash imports to lodash-es which is pretty straight forward thing with Babel plugin, it's impossible to do with current plug-in system.

I understand why, as it's affects performance a lot.

Is there Go API for such plug-ins? So I, or somebody else, can go and write some that I want and then somehow extend esbuild with it?

Thanks :)

[evanw]

You're correct that there is no AST plugin API. I'm not planning on including one due to the performance impact.

However, you shouldn't need one to modify import paths. You should be able to use an on-resolve plugin to intercept the path resolution of the import for lodash and return the path to lodash-es instead.

[RIP21]

@evanw great, thanks for info, didn't knew that. It was just a small example tho, not the main question :)

My main concern was is something more advanced tho. Like Babel plug-ins such as emotion babel plugin, styled-components etc. that do certain optimizations to the code that is using those libraries. Will it be possible in the future or now, to implement those using Golang API to keep performance, while support more optimizations + handle corner cases, like esbuild-jest, which is currently not possible due to ESBuild being very correct to ES spec making it impossible to mock things - hence only suitable for functional integration tests with 0 mocks :)

[zaydek]

@RIP21 From my take of it, see (#821), theoretically but it’s not a supported workflow to interop JS plugins (I gather that’s what you’re talking about) with esbuild in Go. I could be wrong, this is just my current understanding.

[RIP21]

@RIP21 From my take of it, see (#821), theoretically but it’s not a supported workflow to interop JS plugins (I gather that’s what you’re talking a bout) with esbuild in Go. I could be wrong, this is just my current understanding.

I'm willing to learn enough Go to build various plug-ins that will work with AST directly in Go runtime hence quick and fast. That's why I'm asking if there will be plugin system but only for Go native AST transforms that will be quick and performance hit will be none or reasonable.

I'm unsure how Go works internally, so, I guess dynamic imports like in JS may be impossible in Go to integrate such things seamlessly.

[longlho]

hello 👋🏼 just wanna add our use case here for the AST plugin API. I maintain react-intl/formatjs and we do have babel plugin & TS AST transformer to precompile ICU messages since runtime parsing is pretty costly. Therefore we'd love to support esbuild as well.

[DylanPiercey]

Would it be possible to add (async) hooks for build start and build end? I'm thinking similar to rollups buildStart and buildEnd hooks.

Right now we have setup, but it's not async. My particular use case is actually doing some async file system calls before augmenting the input options via a build start hook. A buildEnd hook would be primarily useful for things like clearing caches, cleanup, etc, especially in watch mode with rebuilds.

[evanw]

I can see needing async setup to modify build options. I can add that.

I have already been planning to add callbacks for buildStart and buildEnd. I just haven't done this yet. However, I don't think buildStart needs to be async because build options are not able to be modified at that point. Any async operation can just be awaited in the other callbacks instead. That way asynchronous buildStart logic doesn't block the build or other plugins.

[DylanPiercey]

That sounds great. For my current use case I was planning on making the inputOptions be derived from a file on disk, and so it would be nice if it was possible to return watchFiles like other hooks. My current understanding is that setup is invoked multiple times in watch mode, for each build, is it possible to mutate the input options during watch mode? Does that cause any issues?

Also I’m wondering at what point you imagine buildEnd would be called? Will there eventually be hooks to be able to work on the output files? For example running gzip, uploading to cdn, etc?

[evanw]

My current understanding is that setup is invoked multiple times in watch mode

That is incorrect. The setup function is only run once for the first full build, but not for any of the following incremental builds triggered by watch mode.

Also I’m wondering at what point you imagine buildEnd would be called? Will there eventually be hooks to be able to work on the output files? For example running gzip, uploading to cdn, etc?

I plan for it to get the final BuildResult and to run right before the final result is returned. One thing that still needs a decision is what to do about errors during that phase since the log has already ended.

[DylanPiercey]

I plan for it to get the final BuildResult and to run right before the final result is returned. One thing that still needs a decision is what to do about errors during that phase since the log has already ended.

That sounds good. Is the plan to have it so that the build result can be mutated via this hook?

Also I am wondering if these two use cases I have would be able to be tackled via one of these new build hooks or what your thoughts are.

1. Being able to dynamically change the entryPoints config at any time. Especially during incremental builds. The idea being that if you are serving a multi page app you could lazily discover or add dependencies during the watch lifecycle.

2. Being able to output precompressed (.gzip, .br) versions of the assets. This one could likely be taken care of via a buildEnd hook with the BuildResult being available, but I wonder if that one would be common enough to warrant just being an API option.

[c58]

@evanw any reason why the onResolve of a plugin executed sequentially for every import from one file? I'm writing my own bundler on top of esbuild and I need to resolve every single import via a plugin. When I tried to bundle @material-icons/icons i found that it resolves every import from a long list of icon imports (https://cdn.jsdelivr.net/npm/@material-ui/icons@4.11.2/esm/index.js) one by one, which is very slow. I believe that resolving them in parallel will speed up the process significantly. What do you think?

[glen-84]

You're correct that there is no AST plugin API. I'm not planning on including one due to the performance impact.

Is this your final decision? I'm wanting to switch to Vite, but as mentioned above, FormatJS requires an AST. 😞

[evanw]

any reason why the onResolve of a plugin executed sequentially for every import from one file?

The vast majority of source files only have a few imports, so this has never come up. Having over 5,000 imports in one file is an extreme edge case. Parallelizing things can have overhead so you have to be careful when adding more parallelism. I'm willing to parallelize this if it's an improvement for this edge case if and only if it doesn't regress performance for common scenarios at all.

Is this your final decision?

Yes. You are still welcome to transform the JavaScript code with other tools before or after esbuild runs of course.

[shellscape]

Another vote for extending esbuild's capabilities. I'm bundling lambdas with CDK, and the app imports .graphql files via graphql-import-node. Unfortunately this fails, and I've got some additional hoops to jump through for ESBuild compatibility. It looks like this https://github.com/luckycatfactory/esbuild-graphql-loader will do the trick. Is this issue just stale and out of date?

[mohsen1]

When not bundling, plugin system is not invoked at all. In my use-case I want to modify some import specifiers from our internal convention to something that works in node runtime. A simple plugin does the job but I noticed that the plugin's onResolve is not invoked when bundle is set to false. I can't use bundling because this node.js codebase uses fs heavily to dynamically load files etc.

[eric-hemasystems]

I see the comment above about no AST, but is there any possibility for a plugin hook that allows a custom JS transformations to be applied?

It doesn't need to be handed the AST. Just the final JS after all the transformations that esbuild has applied. The hook is given that JS (and sourcemap) and whatever JS it returns (possibly with an updated sourcemap) is included in the bundle (or passed onto the next plugin that also adds a transformation).

This way if the output of a plugin is JavaScript code and indicates the JS loader then the transformation can be applied even if the original source was not a JS file (for example a Vue component).

My need here is driven from the trying to add code coverage instrumentation via Istanbul. I can currently use the esbuild-babel plugin in combination with the babel-plugin-istanbul plugin. But since esbuild-babel hooks into onLoad it only see the JS on disk not any JS generated from another plugin (like the esbuild-vue plugin). If such a hook existed to add a custom transformation then esbuild-babel could be modified to operate on this new hook instead of onLoad and therefore could apply to all JS generated by another plugin and not just JS on disk.

For more background (as well as my hack workaround) see https://github.com/marvinhagemeister/karma-esbuild/issues/33#issuecomment-1093042415

[intrnl]

647 proposes onTransform hooks that runs after onLoad but before any transformation by esbuild itself

[ZimNovich]

@evanw, Is there a way for a JS plugin to further transform the code after it has been transformed by esbuild itself, but before the file is finally written to disk? I need to make some changes to the code after it has been converted by esbuild from TypeScript to JS. Thank you!