Open Jarred-Sumner opened 2 years ago
I made a test repo that exercises a number of things that are currently having problems with the bun napi interface:
https://github.com/kriszyp/bun-test-apw
Clone that, npm install
it, and then you can compare node index.js
to bun index.js
:
napi_call_threadsafe_function
napi_release_threadsafe_function
not triggering thread_finalize_cb
NAPI_MODULE
macro fails with symbol 'napi_register_module_v1' not found in native module
(easy workaround, since NAPI_MODULE_INIT works fine)You will have uncomment NAPI_MODULE from apw.cpp (and comment NODE_MODULE_INIT) when you want to test that, figured the other things were more important.
Hope that helps, thanks for the great work!
I also wanted to check to see if this is roughly the right approach loading a module for napi + ffi usage. I think the hybrid strategy is good: use napi for general purpose stuff like building objects, async work, and use ffi for "hot" functions that need to be fast. Here is an example from my code where I am loading the platform's binary with the napi loading package, and then, if running in bun, loading it through the FFI loader. But, I think this raises a couple points:
require.resolve
doesn't seem to exist in bun, and this seems challenging for loading a platform-specific package with a binary, which is direction that we are going with node-gyp-build: https://github.com/prebuild/node-gyp-build/pull/45/files#diff-e727e4bdf3657fd1d798edcd6b099d6e092f8573cba266154583a746bba0f346R50Anyway, here is my loading sequence right now:
import { dirname, join, default as pathModule } from 'path';
import { fileURLToPath } from 'url';
import loadNAPI from 'node-gyp-build'; // loads the correct napi binary, using the current OS/arch
let dirName = (typeof __dirname == 'string' ? __dirname : // for bun, which doesn't have fileURLToPath
dirname(fileURLToPath(import.meta.url))).replace(/dist$/, ''); // for node, which doesn't have __dirname in ESM
export let nativeAddon = loadNAPI(dirName);
if (process.isBun) {
const { dlopen, FFIType } = require('bun:ffi');
// load the same napi module as above, but use bun's ffi
// ideally we should use require.resolve to find a package, but here just hoping it is a sibling package in node_modules
let libPath = join(dirName, '..', 'lmdb-linux-' + process.arch, 'node.napi.node');
let lmdbLib = dlopen(libPath, {
getByBinary: { args: [FFIType.f64, FFIType.u32], returns: FFIType.u32},
iterate: { args: [FFIType.f64], returns: FFIType.i32},
position: { args: [FFIType.f64, FFIType.u32, FFIType.u32, FFIType.u32, FFIType.f64], returns: FFIType.i32},
write: { args: [FFIType.f64, FFIType.f64], returns: FFIType.i32},
resetTxn: { args: [FFIType.f64], returns: FFIType.u8},
});
Object.assign(nativeAddon, lmdbLib.symbols);
}
let dirName = (typeof dirname == 'string' ? dirname : // for bun, which doesn't have fileURLToPath
For bun you can also do import.meta.dir
and/or import.meta.path
, which at least saves you from the typeof
check
I assume calling dlopen twice (once through the napi loader, once through the ffi loader) is cached at some level of the OS and doesn't result in duplicate binary footprints in memory
This is a good point. I don't know either, but worth checking.
require.resolve doesn't seem to exist in bun, and this seems challenging for loading a platform-specific package with a binary, which is direction that we are going with node-gyp-build: https://github.com/prebuild/node-gyp-build/pull/45/files#diff-e727e4bdf3657fd1d798edcd6b099d6e092f8573cba266154583a746bba0f346R50
This is an oversight – require.resolve should exist but does not currently. There are at least 5 other ways you can resolve without loading modules though:
Bun.resolve(path, fromPath)
(async)Bun.resolveSync(path, fromPath)
import.meta.resolve(path)
(async)import.meta.resolveSync(path)
(sync)assume calling dlopen twice (once through the napi loader
My evidence is that when I set the value of a static C variable from a NAPI call, that value was accessible/correct from an FFI call, so I think working correctly, but I am not claiming this as absolute proof.
This is an oversight – require.resolve should exist but does not currently
Ok, so you plan on adding require.resolve? I think in node-gyp-build it would either need to go through createRequire(libraryPath)
, or use the two arg version (require.resolve(id, { path: [ libraryPath ] }
). node-gyp-build
is CJS module, so unfortunately import.meta.resolve
can't be used there (and can't even be mentioned, if (false) import.meta.resolve('test')
will even fail in Node in CJS mode).
Anyway, thanks for the progress here, I know these are probably annoying details of the napi/node machinery, but I think node-gyp-build is pretty commonly used.
Ok, so you plan on adding require.resolve? I think in node-gyp-build it would either need to go through createRequire(libraryPath), or use the two arg version (require.resolve(id, { path: [ libraryPath ] }). node-gyp-build is CJS module, so unfortunately import.meta.resolve can't be used there (and can't even be mentioned, if (false) import.meta.resolve('test') will even fail in Node in CJS mode).
Support for both of these was added in Bun v0.0.83 (released today)
FYI, I tried to run parcel-css with Bun, but got the following error (logging with DYLD_PRINT_APIS
). Not sure if this is useful. I tried to debug with lldb but wasn't able to due to macOS code signing. I guess I'd have to compile bun myself to debug further.
dyld[50777]: dlopen("/Users/devongovett/dev/parcel-css/parcel-css.darwin-arm64.node", 0x00000001)
dyld[50777]: dlopen(parcel-css.darwin-arm64.node) => 0x2089957e0
dyld[50777]: dlsym(0x2089957e0, "napi_register_module_v1")
dyld[50777]: dlsym("napi_register_module_v1") => 0x110ab41b0
dyld[50777]: dlsym(0xfffffffffffffffe, "getentropy")
dyld[50777]: dlsym("getentropy") => 0x183fd8f5c
Example:
const parcelCss = require('./parcel-css.darwin-arm64.node');
parcelCss.transform({
filename: 'test.css',
code: Buffer.from('.foo { color: red }'),
minify: true
});
If I attempt to load a native module that uses Node-API, I get a crash on missing symbols. Additionally, I don't see any of the Node-API symbols in the bun binary (0.1.1, macOS x86_64).
Is there something additional that needs to be done to load these symbols or was this possibly optimized out in the latest release?
$ symbols $(which bun) | grep napi
Edit: It appears to be only the macOS release where they are missing. The symbols are present in linux.
What do you mean by async_hooks resource tracking will not be supported, it will just ignore them.
As bluebird uses async_hooks and requires it explicitly like this for example:
var AsyncResource = util.isNode && util.nodeSupportsAsyncResource ?
require("async_hooks").AsyncResource : null;
Which then results into:
Could not resolve: "async_hooks". Maybe you need to "bun install"?
async_hooks will need to be polyfilled but initially just disabled so it is a no-op rather than failure on require()
Also in bun at least, you should try to avoid Bluebird. It’s a many X slowdown
On Sun, Jul 10, 2022 at 10:13 AM Laurens Lavaert @.***> wrote:
What do you mean by async_hooks resource tracking will not be supported, it will just ignore them.
As bluebird uses async_hooks and requires it explicitly like this for example:
var AsyncResource = util.isNode && util.nodeSupportsAsyncResource ? require("async_hooks").AsyncResource : null;
Which then results into:
Could not resolve: "async_hooks". Maybe you need to "bun install"?
— Reply to this email directly, view it on GitHub https://github.com/oven-sh/bun/issues/158#issuecomment-1179765632, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFNGSZC646YQEGC6NGCNRTVTMAFNANCNFSM5VDV6N3Q . You are receiving this because you authored the thread.Message ID: @.***>
Also in bun at least, you should try to avoid Bluebird. It’s a many X slowdown
What would you say is recommended to use (instead)? Par example,
(async () => { for (let i=0; i<800000; i++) { await functionCall(i); } })();
...would work, but only one execution at a time.
(And without limitation program will produce errors.)
(Promise.map([...Array(800000).keys()], functionCall, {concurrency: 8});)
P. S. http://bluebirdjs.com/docs/benchmarks.html claims to be faster (in node) than async-await. P. S. If I am not mistaken, another problem rises with big arrays: bun took 7 GB (and counting) of RAM for 800 000 keys. Speaking of slowdowns.
PrismaJS also requires async_hook
Hey folks! Is there any progress on napi_create_external
?
Hey folks! Is there any progress on
napi_create_external
?
Added in https://github.com/oven-sh/bun/commit/dddbce8a41aa266af106b19e583c152126900730
When running @parcel/watcher, even just trying to import it with import watcher from '@parcel/watcher'
bun fails with Segmentation fault: 11
(that's all it outputs). Since that module is written in C++ I figured it might be NAPI-related, might be related to this issue?
When running @parcel/watcher, even just trying to import it with import watcher from '@parcel/watcher' bun fails with Segmentation fault: 11 (that's all it outputs). Since that module is written in C++ I figured it might be NAPI-related, might be related to this issue?
Yes, there are at least three issues blocking @parcel/watcher
from working, of which I just fixed two:
Looks like napi_set_instance_data()
might be supported (I have seen some cpp impl in the repo) but it's probably not exported. I get this on my macos. Here's repro if anyone wants to have a look.
https://github.com/cztomsik/zig-napigen/tree/main/example
@cztomsik Yep, that was a mistake. Please try again once the canary build finishes (bun upgrade --canary
)
229f5f7770248cee57ee6b9c021c348cca8b6bbb
@Jarred-Sumner Either I did something wrong or did not help :-/
Sorry, please try again after 4be3548829d6083cfe7c488eb84e310b952b88e1
This time it should work
Works great, thanks!
Hello, I'm doing some examples of add-ons with ziglang. Bun works wonderfully with the following examples:
It breaks, 'core dumped', with the following equivalent examples:
cd node-addon-example-async-work-thread-safe-function-zig
# Bun
bun example.js
Segmentation fault (core dumped)
# With nodejs
node example.js
Received prime from secondary thread: 7919
Received prime from secondary thread: 17389
Received prime from secondary thread: 27449
...
cd node-addon-examples/async_work_thread_safe_function/napi
# Bun
bun: ../binding.c:231: napi_register_module_v1: Assertion `status == napi_ok' failed.
Aborted (core dumped)
# With nodejs
node index.js
Received prime from secondary thread: 7919
Received prime from secondary thread: 17389
Received prime from secondary thread: 27449
...
Bun version
bun --version
0.6.5
Nodejs version
node --version
v20.2.0
Does it have a known cause? Many thanks for your work.
NPM module sodium-native
is not working due to undefined exports. My understanding of C and N-Api is too limited to understand what is really happening, but the module seems to use only n-api features that are marked as supported in the support list at the head of this issue.
Bug demonstrated here #3652
This is not working yet:
But this other new version works with bun:
Bun version
bun --version
0.7.0
Nodejs version
node --version
v20.5.0
Having a napi_register_module_v1 related error with ganache as well:
> bun src/gabacge_test.ts
bigint: Failed to load bindings, pure JS will be used (try npm run rebuild?)
This version of µWS is not compatible with your Node.js build:
TypeError: symbol 'napi_register_module_v1' not found in native module. Is this a Node API (napi) module?
Falling back to a NodeJS implementation; performance may be degraded.
(stack trace)
^
error: Premature close
code: "ERR_STREAM_PREMATURE_CLOSE"
at new NodeError (node:stream:2:5616)
at node:stream:2:48388
at emitCloseNT (node:stream:2:72318)
Example:
import ganache from 'ganache';
const options = {
fork: 'https://bsc-dataseed.binance.org'
}
const server = ganache.server(options);
server.listen(port, async (err) => {
if (err) throw err;
console.log(`ganache listening on port ${port}...`);
});
The above mention from an issue in keccak is interesting:
But using bun:
The build succeeds and the node-gyp-build require succeeds...... returning an empty object {}
instead of a function like it should.
So our fallback mechanism doesn't work with bun and users get cryptic errors (see the issue).
It would be nice if bun could at least throw an error when it tries to build / run something it can't support, that way our existing mechanism would have caught the issue.
Let me know if this should be a separate issue. Thanks.
Bun as of v1.0.3 imports Napi functions as object (in node: typeof x -> 'function', in bun: typeof x -> 'object').
This renders the imported Napi function useless.
For me it worked to recompile myNapi code with an object returned:
exports.Set('myfunc
, ....)insted of
exports = ...`.
Of course such a recompile (actually involving a small redesign) is not always feasable.
It should be treated as bug and eventually fixed.
Another compatibility request: cpu-features
, which is used transitively by libraries like dockerode
and node-ssh
via ssh2
. Importing cpu-features
currently fails with the following error about missing symbols:
dyld[60820]: missing symbol called
[1] 60820 killed bun index.mjs
AFAICT this is an NAPI issue; let me know if that's not the case. This is blocking us from adopting Bun in a variety of our projects.
@nwalters512 cpu-features
appears to use nan
which links directly to the V8 headers instead of the Node-API stable C headers.
The cpu-features
maintainers would need to update their library to use Node-API in order to be compatible with bun
.
@kjvalencik ah, that's unfortunate, thanks for looking into that!
This is probably a difficult ask, but is it possible to make require('cpu-features')
fail with a Node error instead of crashing the process? Most packages that use cpu-features
wrap its require in try
/catch
so they can proceed even if it isn't available. This would allow Bun to continue executing, albeit without access to cpu-features
, which is tolerable in a lot of cases.
Hey guys! Any plans to support napi_env
and napi_callback_info
? I'm trying to use @elgato-stream-deck/node
, but I'm receiving a segfault.
Any updates on this?
Support resolving .node files
doesn't seem to work with the --compile
flag, bun
will report a successful build but no binary will be produced
i got the error at uWS.js also
TypeError: symbol 'napi_register_module_v1' not found in native module. Is this a Node API (napi) module?
@TomieAi uWebSockets.js
isn't a Node-API addon.
@TomieAi
uWebSockets.js
isn't a Node-API addon.
hmm i know but i just wanna say uws dint work at bun sadly. throwing me error about "napi_register_module_v1" then i google about it XD and it leads me here.
@TomieAi That issue would need to be filed with uWebSockets.js. This isn't an issue with bun
. It's unlikely that bun
will support native addons that don't use Node-API since they generally interact directly with V8 (which bun doesn't use).
For anyone experiencing NAPI issues with bun
and the sqlite3
npm
package, there is a built-in package available:
My issue regarding napi-nanoid
or any other rust based napi packages
Node-API is Node.js' native addon API. The goal of this project is for many Node-API addons in bun.js to just work, without asking maintainers of node modules to make changes or rebuild their code specifically for Bun.
The first version will be scoped specifically to the
napi_*
functions. It will not include support foruv_*
functions or the V8 C++ functions. A good initial target is napi.rs support.Internal changes
.node
filesprocess.dlopen
support for native modulesrequire
support for native modulesAPI implementation
JSC::VM::lastException()
)Test coverage
The tentative plan is to rely on napi.rs tests, but haven't checked yet if that's possible.