Closed sessionboy closed 3 years ago
What's stopping you from using esbuild with deno? I've been doing so happily for a while.
Also, I've actually been working with people on making Node features work in Deno using esbuild in the past.
@benjamingr Hi, is there any sample code? I cannot import esbuild to use in deno. esbuild uses the code of the node core module, which cannot be executed in deno. E.g:
Uncaught ReferenceError: require is not defined
What is the way you use esbuild in deno?
I know a way:
const p = Deno.run({
cmd: ["node", "./esbuild.js"],
stdin: "piped",
stdout: "piped",
stderr: "piped",
});
But this requires you to install node and write node code, and its compilation speed will also slow down.
I hope I can directly import esbuild in deno and use it:
import esbuild from 'https://esm.sh/esbuild';
esbuild is written in Go, the Node.js wrapper just calls into it it's a native module - see https://github.com/evanw/esbuild/tree/master/npm - you can technically probably use the WASM version or build a native Deno module - but you can also just grab the binary and run the command in another process.
Right but you can't run plugins via the binary alone. You need the Node library (here: https://www.unpkg.com/browse/esbuild@0.9.0/lib/main.js) which uses require
and a few Node-specific modules like "fs" and "child_process". Porting the code to Deno wouldn't be too hard, especially if you skip the installation code and just leave the esbuild binary in your project root (since there's no node_modules
folder in Deno).
Edit: You might also be able to use the Node compat module from Deno's std library... Not sure.
Are those APIs not compatible with std/node?
you can technically probably use the WASM version or build a native Deno module - but you can also just grab the binary and run the command in another process.
@benjamingr I know, the official document mentions this. https://esbuild.github.io/getting-started/#other-ways-to-install.
But at the same time it also shows that the wasm version is slow. The binary esbuild does not support plugins, i need plugin.
which uses require and a few Node-specific modules like "fs" and "child_process". Porting the code to Deno wouldn't be too hard
@heyheyhello Maybe this is a good way. But I still hope that esbuild can natively support deno.This way I can use its full features without affecting the speed.
Give it a shot with Deno's std/node. @benjamingr is probably right - the compatibility layer should work.
@heyheyhello I checked your link, this is the node module referenced by esbuild:
// lib/node.ts
--
1274 | var child_process = require("child_process");
1275 | var crypto = require("crypto");
1276 | var path = require("path");
1277 | var fs = require("fs");
1278 | var os = require("os");
1279 | var tty = require("tty");
I compared std/node, https://github.com/denoland/deno_std/tree/0.90.0/node, these modules are still not implemented or only partially implemented in std/node. So still need to wait...
Deno uses cached HTTP imports. If you just want to bundle a Deno project, you can use esbuild-plugin-cache with the Node API. This plugin uses the same algorithm as Deno to cache, which means that the HTTP modules that you have cached during the development will not be download again during the bundle. The plugin also allows esbuild to use import-maps, which are now stable in Deno.
So I'm very unfamiliar with Deno. I have literally never used it before. However, I just tried it out now and I was able to get it to work with this code:
import * as esbuild from 'https://esm.sh/esbuild-wasm/esm/browser.js'
await esbuild.initialize({
wasmURL: 'https://esm.sh/esbuild-wasm/esbuild.wasm',
worker: false,
})
console.log(await esbuild.transform('1+2+3'))
It looks like you have to use worker: false
because otherwise I get this error:
Uncaught (in promise) Error: Not implemented
at Function.createObjectURL (deno:op_crates/web/11_url.js:394:13)
Apparently URL.createObjectURL()
doesn't work? How strange. It looks like they deliberately broke it but I don't understand why: https://github.com/denoland/deno/pull/7543. Anyway, at least it works. Passing worker: false
just means that it will run on the main thread instead of in a background thread.
With that you should be able write some esbuild plugins that interact with the file system using Deno's file system APIs. It would be interesting to see someone make a proof of concept for this.
As far as running the native version of esbuild in Deno, I assume the code to do this would need to download and unzip the appropriate binary executable from npm at run-time. Is that correct? Am I right that Deno doesn't have a notion of install scripts? If so, is there a standard way to do this in Deno? For example, I'm guessing that Deno has some form of HTTP download cache already built in if you import modules by URL.
Then you could imagine esbuild having support for Deno by doing something like import "https://esm.sh/esbuild-wasm/deno/index.js"
and having that automatically download and unzip the esbuild binary and store it to some appropriate place locally. Basically what the current node install script does.
I'm not saying I'm definitely going to support Deno in esbuild as a first-class citizen. I'm just trying to think through what a potential implementation could look like.
So I'm very unfamiliar with Deno
Deno is very easy to use, I believe you can learn and use it within a few hours.
import * as esbuild from 'https://esm.sh/esbuild-wasm/esm/browser.js'
const t = performance.now()
await esbuild.initialize({
wasmURL: 'https://esm.sh/esbuild-wasm/esbuild.wasm',
worker: false,
})
console.log(await esbuild.transform('1+2+3'))
console.log(`compile in ${Math.round(performance.now() - t)}ms`)
@evanw The esbuild-wasm does work, but it is very slow. The above build took 2000ms, but the node's native version is less than 20ms, which is a difference of 100 times.
Am I right that Deno doesn't have a notion of install scripts?
Yeah, you are right. Deno does not have node_modules, it will be downloaded from the url and cached locally. The url does not point to a binary file, but the actual code. E.g: https://esm.sh/react (You can open it in the browser address bar.)
As far as running the native version of esbuild in Deno, I assume the code to do this would need to download and unzip the appropriate binary executable from npm at run-time. Is that correct?
I haven't tried it, I'm not sure. But as you describe in the documentation(https://esbuild.github.io/getting-started/#download-a-build):
Why this is not recommended: This relies on internal implementation details of esbuild's native executable installer. These details may change at some point, in which case this approach will no longer work for new esbuild versions. This is only a minor drawback though since the approach should still work forever for existing esbuild versions since packages published to npm are immutable. An additional drawback is that you cannot use plugins with the native version.
Deno follows the esm specification, and any code that conforms to the esm specification can be run in deno. The reason why esbuild cannot run directly in deno is that esbuild uses the following node core modules:
// esbuild
// lib/node.ts
--
1274 | var child_process = require("child_process");
1275 | var crypto = require("crypto");
1276 | var path = require("path");
1277 | var fs = require("fs");
1278 | var os = require("os");
1279 | var tty = require("tty");
These node modules follow the commonjs specification and are not compatible with deno.Deno is trying to be compatible with node core modules, but it is still in progress.https://github.com/denoland/deno_std/tree/0.90.0/node.
I'm not saying I'm definitely going to support Deno in esbuild as a first-class citizen. I'm just trying to think through what a potential implementation could look like.
I think the best solution is to include deno compatibility code in esbuild code so that esbuild can run in node and deno at the same time. There are currently many libraries that do this. E.g: https://github.com/cacjs/cac . I use it in node, and now I also use it in deno, there is no obstacle.
// in node
import { cac } from 'cac'
// in deno
import { cac } from 'https://unpkg.com/cac/mod.ts'
@evanw The esbuild-wasm does work, but it is very slow. The above build took 2000ms, but the node's native version is less than 20ms, which is a difference of 100 times.
I assume that this delay isn't due to Deno re-downloading everything every time. Is there a way to verify that? Given that, I assume that the overhead is due to WebAssembly compilation. Which is interesting because in theory this doesn't have to be the case. The way WebAssembly is designed, it is pretty straightforward to compile and cache the translation of the WebAssembly module to assembly so that repeated invocations can skip the compilation time completely and be very fast.
V8 actually implements this: https://v8.dev/blog/wasm-code-caching. In the browser, triggering this involves using WebAssembly.instantiateStreaming
instead of WebAssembly.instantiate
. Currently esbuild uses WebAssembly.instantiate
. I wonder if changing that would give a massive speedup in Deno. Deno would presumably need to use V8's code caching API to have this work.
I think the best solution is to include deno compatibility code in esbuild code so that esbuild can run in node and deno at the same time. There are currently many libraries that do this. E.g: https://github.com/cacjs/cac . I use it in node, and now I also use it in deno, there is no obstacle.
I wanted to explore what it would take to do this as a thought experiment. I think the way to do this would have to be to have a separate entry point file (or equivalently a separate package) for Deno because esbuild's node integration is very specific to node. The entry point file would also have to be an installer, which further differentiates it from the node entry point. Deno's compatibility with node's packages is irrelevant with that approach.
I assume that this delay isn't due to Deno re-downloading everything every time. Is there a way to verify that?
@evanw Deno will cache all downloads, unless you specify --reload
it will download again. I didnβt study wasm in depth, so I donβt know. You have described this problem in the esbuild documentation(https://esbuild.github.io/getting-started/#other-ways-to-install):
Why this is not recommended: The WebAssembly version is much, much slower than the native version. In many cases it is an order of magnitude (i.e. 10x) slower. This is for various reasons including a) it's compiled from scratch on every run, b) the WebAssembly compilation approach is single-threaded, and c) node has WebAssembly bugs that can delay the exiting of the process by many seconds. The WebAssembly version also excludes some features such as the local file server. You should only use the WebAssembly package like this if there is no other option, such as when you want to use esbuild on an unsupported platform. The WebAssembly package is primarily intended to only be used in the browser.
I think the way to do this would have to be to have a separate entry point file (or equivalently a separate package) for Deno because esbuild's node integration is very specific to node
You are right, as cac did. Deno does not need to be installed, just provide an entry file for download. Like the deno entry point file of cac : source https://github.com/cacjs/cac/blob/master/mod.ts esm: https://unpkg.com/cac/mod.ts . I don't think it will be very difficult. There are not many node core libraries actually used in esbuild. Cheers π»
V8 actually implements this: https://v8.dev/blog/wasm-code-caching. In the browser, triggering this involves using
WebAssembly.instantiateStreaming
instead ofWebAssembly.instantiate
. Currently esbuild usesWebAssembly.instantiate
. I wonder if changing that would give a massive speedup in Deno. Deno would presumably need to use V8's code caching API to have this work.
I just tried this out. I couldn't get it to work with local files because a) Deno deliberately uses the file
URL scheme for local files and b) fetch
deliberately doesn't work with the file
URL scheme: https://github.com/denoland/deno/issues/2150 and because c) they deliberately broke URL.createObjectURL
: https://github.com/denoland/deno/pull/7543. I think that means it's impossible to use WebAssembly.instantiateStreaming
for local files. This is a shame because it could potentially be a massive performance win for WebAssembly in Deno (the WebAssembly code cache, not WebAssembly.instantiateStreaming
).
Sad...o(β₯οΉβ₯)o
b) fetch deliberately doesn't work with the file URL scheme: denoland/deno#2150.
@evanw They seem to be solving this problem. Although I don't know why you need to use fetch to read local files, maybe you can try to read directly with readFileSync.
Deno.readFileSync(new URL("file:///Volumes/work/deno-test/index.ts").pathname)
Or set up a static server locally and request local files through http. If it is not under the same domain name, you can use a proxy:
// ./static-server.ts
import { Application, send } from "https://deno.land/x/oak/mod.ts";
const app = new Application();
// set static dir
app.use(async (context) => {
await send(context, context.request.url.pathname, {
root: Deno.cwd(),
});
});
await app.listen({ port: 8000 });
// ./fetch.ts
const content = await fetch(`http://localhost:8000/test.ts`).then(res=res.text());
console.log(content);
c) they deliberately broke URL.createObjectURL: denoland/deno#7543.
This is really bad, but it is mentioned in their Q1 2021 roadmap
to solve this problem (#9210).
blob: URL support (#9210)
Q1 2021 roadmap: https://github.com/denoland/deno/issues/8824
I think that means it's impossible to use WebAssembly.instantiateStreaming for local files. This is a shame because it could potentially be a massive performance win for WebAssembly in Deno (the WebAssembly code cache, not WebAssembly.instantiateStreaming)
I donβt know, maybe we can send them a pr.
I am kind of walking in here with little context, but I think it should be noted that Deno has a native plugin API for Rust - maybe you could bind the Rust to the Go then have it use the native plugin API.
Just wondering, any reason you are not using deno bundle
?
Hey π, Deno maintainer here.
I assume that this delay isn't due to Deno re-downloading everything every time. Is there a way to verify that? Given that, I assume that the overhead is due to WebAssembly compilation. Which is interesting because in theory this doesn't have to be the case. The way WebAssembly is designed, it is pretty straightforward to compile and cache the translation of the WebAssembly module to assembly so that repeated invocations can skip the compilation time completely and be very fast.
Actually it is due to repeated downloads. There is no cache on fetch
, so the WASM is re-fetched on each invocation. A possible solution for this would be to provide a wasmData
init option instead of wasmUrl
that would take an ArrayBuffer or typed array. Users could then use something like https://deno.land/x/cache to cache the download. Hopefully this pain point will be reduced once we get support for WASM modules (using import assertions), as WASM can then be part of the module graph.
it's impossible to use WebAssembly.instantiateStreaming for local files
Not quite - you can do this, but it requires some trickery. You can open the file with Deno.open
, then convert the returned Reader into a WHATWG ReadableStream using https://doc.deno.land/https/deno.land/std/io/streams.ts#readableStreamFromIterable, and then convert the ReadableStream
into a Response
which can then be passed into WebAssembly.instantiateStreaming
. This is all theoretical though, because our WebAssembly.instantiateStreaming
is broken anyway: https://github.com/denoland/deno/issues/7259.
Regarding native esbuild using the esbuild binary: I think the most straight-forward way would be to indeed download, unzip, and cache the binary at runtime (again you could use https://deno.land/x/cache for this), and then start the subprocess with Deno.run
. The only thing that would differ between Node and Deno here would be the subtleties of the sub-process APIs.
Please ping me if there are any more questions :-)
I just published an experimental Deno package for esbuild version 0.11.10. Using it looks like this:
import * as esbuild from 'https://deno.land/x/esbuild@v0.11.10/mod.js'
const ts = 'let hasProcess: boolean = typeof process != "null"'
const result = await esbuild.transform(ts, { loader: 'ts', logLevel: 'warning' })
console.log('result:', result)
esbuild.stop()
It has basically the same API as the npm package with one addition: you need to call stop()
when you're done. Please try it out and let me know what you think. Caveat: I have done very little testing, and I have only tested this on macOS and Linux. It should hypothetically have all of the features of the node package because it's a very straightforward port, but I haven't tested it much yet. I'll be doing more testing later but I wanted to get it out early for feedback.
Please ping me if there are any more questions :-)
Thanks for reaching out @lucacasonato. It's good to have additional detail about the WebAssembly implementation. I'm going to focus on the native approach given that it's much faster and has more features than the WebAssembly version. I didn't end up using https://deno.land/x/cache because the file I need to cache is inside the downloaded file and needs to be extracted first. You can see my current implementation here. I'm just writing the same information to the same cache directory as my npm package installer.
The one big hiccup that I hit when porting my node-based implementation to Deno is that I didn't see any equivalent to node's ref()
and unref()
functionality on a child process. I'm looking for a way to allow Deno to exit while a child process is still running. Is doing that possible with Deno? The node-based esbuild API is nice to use because you don't have to worry about managing the lifetime of esbuild's child process as it's automatically managed for you with ref()
and unref()
as you use esbuild's API. I have hacked around this in Deno for now by adding a stop()
function that you have to call manually, but it'd be great to not need to do that in Deno.
I have confirmed that it works on x86_64 macOS and Linux but not on Windows or on ARM macOS. Can someone with either of those two platforms confirm whether it works or not?
I confirm it is working on Windows 10. I tested using this example: https://github.com/dalcib/esbuild-plugin-cache/tree/master/example-deno
Running @dalcib's example I get this error on my M1 Macbook Air (using deno 1.9.0)
> deno run --allow-env --allow-read --allow-write --allow-net --allow-run server.js
Download https://deno.land/std/http/server.ts
Warning Implicitly using latest version (0.93.0) for https://deno.land/std/http/server.ts
Download https://deno.land/std@0.93.0/http/server.ts
Download https://deno.land/std@0.93.0/http/_io.ts
Download https://deno.land/std@0.93.0/io/bufio.ts
Download https://deno.land/std@0.93.0/async/mod.ts
Download https://deno.land/std@0.93.0/http/http_status.ts
Download https://deno.land/std@0.93.0/io/util.ts
Download https://deno.land/std@0.93.0/textproto/mod.ts
Download https://deno.land/std@0.93.0/async/delay.ts
Download https://deno.land/std@0.93.0/async/deferred.ts
Download https://deno.land/std@0.93.0/async/mux_async_iterator.ts
Download https://deno.land/std@0.93.0/async/pool.ts
Download https://deno.land/std@0.93.0/bytes/mod.ts
Download https://deno.land/std@0.93.0/io/buffer.ts
Check file:///Users/tobias/dev/github/esbuild-plugin-cache/example-deno/server.js
error: Uncaught NotFound: No such file or directory (os error 2)
if (clients.length === 0) Deno.run({ cmd: ['cmd', '/c', 'start', `http://localhost:3000`] })
^
at unwrapOpResult (deno:core/core.js:100:13)
at Object.opSync (deno:core/core.js:114:12)
at opRun (deno:runtime/js/40_process.js:20:17)
at Object.run (deno:runtime/js/40_process.js:104:17)
at file:///Users/tobias/dev/github/esbuild-plugin-cache/example-deno/server.js:52:36
at fire (deno:runtime/js/11_timers.js:443:7)
at handleTimerMacrotask (deno:runtime/js/11_timers.js:303:7)
Also tested deno
v1.9.0 and v1.8.3 on my Windows 10 (build: 21301.1010) machine, where I get these errors:
β example-deno git:(master)β deno run --allow-env --allow-read --allow-write --allow-net --allow-run build.js
Check file:///C:/dev/github/esbuild-plugin-cache/example-deno/build.js
error: Uncaught (in promise) Error: Failed to find cache directory
at getCachePath (https://deno.land/x/esbuild@v0.11.10/mod.js:1460:11)
at installFromNPM (https://deno.land/x/esbuild@v0.11.10/mod.js:1421:33)
at install (https://deno.land/x/esbuild@v0.11.10/mod.js:1497:18)
at https://deno.land/x/esbuild@v0.11.10/mod.js:1510:29
at ensureServiceIsRunning (https://deno.land/x/esbuild@v0.11.10/mod.js:1601:7)
at Module.build (https://deno.land/x/esbuild@v0.11.10/mod.js:1391:26)
at file:///C:/dev/github/esbuild-plugin-cache/example-deno/build.js:11:9
That specific issue βοΈ is fixed (for me and I think all windows machines) by using Deno.env.get("LOCALAPPDATA")
at https://deno.land/x/esbuild@v0.11.10/mod.js#L1446.
I tried adding this line π to the build.js
and server.js
examples before calling esbuild
and running again
Deno.env.set("FOLDERID_LocalAppData", Deno.env.get("LOCALAPPDATA"))
Then server.js
seems to run fine and build.js
produces the file bundle.js
but the process never exits (although I'm not sure if that's the intended outcome or not @dalcib π€· ).
@tolu, thank you for point this out.
a) In my case, I don't have the FOLDERID_LocalAppData system variable defined, and, surprisingly, it worked.
I discovered that it worked because in the absence of the FOLDERID_LocalAppData variable, Esbuild is cached in C:\Users\user\.cache
, according with the code below:
let baseDir;
switch (Deno.build.os) {
case "windows":
baseDir = Deno.env.get("FOLDERID_LocalAppData");
break;
}
if (!baseDir) {
baseDir = Deno.env.get("HOME");
if (baseDir)
baseDir += "/.cache";
}
b) About the stop, you are right. In build.js
the process never stops, even calling esbuild.stop()
in the end.
But I discovered that the location of the esbuid.stop()
was wrong.
I put it in this way now, and it is working properly.
esbuild
.build({
//...
})
.then((result, error) => {
esbuild.stop()
})
c) By the way, the server.js
code for open the browser was written only for Windows.
You can run it on Mac or Linux changing this line:
if (clients.length === 0) Deno.run({ cmd: ['cmd', '/c', 'start', `http://localhost:3000`] })
for it:
const open = { darwin: ['open'], linux: ['xdg-open'], windows: ['cmd', '/c', 'start'] }
if (clients.length === 0) Deno.run({ cmd: [...open[Deno.build.os], 'http://localhost:3000'] })
I already updated it in the repo.
π @dalcib π
a) In my case, I don't have the FOLDERID_LocalAppData system variable defined, and, surprisingly, it worked. I discovered that it worked because in the absence of the FOLDERID_LocalAppData variable, Esbuild is cached in
C:\Users\user\.cache
, according with the code below:let baseDir; switch (Deno.build.os) { case "windows": baseDir = Deno.env.get("FOLDERID_LocalAppData"); break; } if (!baseDir) { baseDir = Deno.env.get("HOME"); if (baseDir) baseDir += "/.cache"; }
βοΈ Great that it worked but I'm just not sure HOME
is a standard on Windows. I can see that git bash
specifically does add that variable but neither CMD, PowerShell or Yori does. So I think LOCALAPPDATA
is the better solution. π @evanw
b) About the stop, you are right. In
build.js
the process never stops, even callingesbuild.stop()
in the end. But I discovered that the location of theesbuid.stop()
was wrong.
π π Worked like a charm (for me on Windows, will test more on M1 Mac)!
So I think LOCALAPPDATA is the better solution.
Yes, I agree with you.
This library also uses LOCALAPPDATA : https://deno.land/x/cache@0.2.12/directories.ts#L28
I don't have the FOLDERID_LocalAppData system variable defined,
Sorry about that. I copied that from the https://deno.land/x/cache_dir/mod.ts
package but I didn't have a Windows machine available for testing. Once it was published, I was able to use GitHub Actions to test on Windows, and I have already discovered and fixed this by using LOCALAPPDATA
. The next release will contain the fix.
Is there an obvious reason I am missing that this doesn't work?
import * as esbuild from "https://deno.land/x/esbuild@v0.11.12/mod.js";
esbuild.build({
entryPoints: ["./src/mod.ts"],
outfile: "out.js",
bundle: true,
format: 'esm',
watch: {
onRebuild(error, result) {
if (error) console.error("watch build failed:", error);
else console.log("watch build succeeded:", result);
},
},
}).then((result, error) => {
esbuild.stop();
});
With deno run --allow-all finalbundle.js
i get watch build failed: Error: The service was stopped at afterClose (https://deno.land/x/esbuild@v0.11.12/mod.js:548:18) at https://deno.land/x/esbuild@v0.11.12/mod.js:1542:11
Yes. You are stopping esbuild as soon as it starts. With watch mode you'll need to keep esbuild running.
I got it working:
import * as esbuild from "https://deno.land/x/esbuild@v0.12.5/mod.js";
esbuild
.build({
entryPoints: ["./src/mod.ts"],
outfile: "./dist/bundle.js",
format: "esm",
bundle: true,
// minify: true,
sourcemap: true,
watch: {
onRebuild(error, result) {
if (error) console.error("watch build failed:", error);
else console.log("watch build succeeded:", result);
},
},
})
.then((result, error) => {})
.catch(() => process.exit(1));
Is there any documentation with examples available on the site, and is this awesome extension becoming 'official' soon?
Here is a working example for 2024:
bundle.ts
Make sure to update entryPoints
and outdir
.
import * as esbuild from "https://deno.land/x/esbuild@v0.20.1/mod.js";
import { denoPlugins } from "jsr:@luca/esbuild-deno-loader@0.9";
esbuild.build({
plugins: [...denoPlugins()],
entryPoints: ["<input>/<dir>/script.ts"],
outdir: "<output>/<dir>/",
bundle: true,
platform: "browser",
format: "esm",
target: "esnext",
minify: true,
sourcemap: true,
treeShaking: true,
});
await esbuild.stop();
deno run --allow-read --allow-write --allow-env --allow-net --allow-run bundle.ts
You can create a deno task
to make running this easier. Add this to your deno.jsonc
:
{
"tasks": {
"bundle": "deno run --allow-read --allow-write --allow-env --allow-net --allow-run bundle.ts"
}
}
esbuild
(git)
esbuild-deno-loader
/denoPlugins
(deno.land/x) (git)
.ts
files include import statements with Deno module specifiers like file:
, https:
, data:
, npm:
, and/or jsr:
.For more context, check out my post on this topic: https://www.toddgriffin.me/blog/how-to-bundle-deno-typescript-for-the-browser
Hi guy! Deno's excellence made me give up node, but I need a build tool like esbuild.
I want to transform npm modules to esm to run in deno, which allows deno to have the entire web ecosystem.
I know swc, but it is not reliable yet, the worst case is that it will transform to wrong code, so I need esbuild, just like node, but not webAssembly(esbuild-wasm).