Node 20.3 Crashes all the time when executed inside docker

[ronag]

• Node 20.3.0 crashes on start in non-bullseye docker container
• In bullseye container node-gyp-build (yarn add bufferutil) fails with Text file busy
• In bullseye container + UV_USE_IO_URING=0 everything works

[santigimeno]

Does this also solve the non bullseye image crash?

No, it's unrelated. That seemed like a docker issue only, right?

[bnoordhuis]

Yes, that's docker's seccomp bug. Fix: upgrade docker.

[ronag]

Yep. I reopenend this issue.

[santigimeno]

Though we should confirm first that the issue is actually fixed

I can confirm the fix works (at least now yarn add bufferutil is installed correctly)

[nick4fake]

I am sorry, so what exactly is solution for now? set UV_USE_IO_URING in production images with possibility that it breaks again when it is removed?

[santigimeno]

The Text file busy should be fixed in the next release that contains libuv v1.46.0 which should be v20.4.0

[Haringat]

Does this still occur in 20.4.0?

[bnoordhuis]

It shouldn't, libuv now blacklists known-bad kernels.

[bnoordhuis]

I'll go ahead and close this now that it's fixed. To summarize:

• node v20.4.0 and newer contain the fix (well, workaround)

• if you still experience issues while running under docker, upgrade docker

[masterkain]

still no official docker image for 20.4, weird

[raxityo]

Unfortunately the PR for the official build is still open https://github.com/docker-library/official-images/pull/14997

[ekzobrain]

Get the same error with image node:18.17.1 and using "ENV UV_USE_IO_URING=0" does not help. But 18.17-alpine works even without UV_USE_IO_URING=0

[santigimeno]

18.17.1 comes with libuv 1.44.2 which does not include the io_uring based implementation. Can you open a new issue describing what you're observing?

[mitar]

I observed the same (similar) issue on v12.22.9 and now I upgraded to v20.8.0 and I still have the same issue. When I run node inside a Docker container, I get (this is on v20.8.0):

node[10]: ../src/node_platform.cc:68:std::unique_ptr<long unsigned int> node::WorkerThreadsTaskRunner::DelayedTaskScheduler::Start(): Assertion `(0) == (uv_thread_create(t.get(), start_thread, this))' failed.
 1: 0xc99960 node::Abort() [node]
 2: 0xc999de  [node]
 3: 0xd198a9 node::WorkerThreadsTaskRunner::WorkerThreadsTaskRunner(int) [node]
 4: 0xd199cc node::NodePlatform::NodePlatform(int, v8::TracingController*, v8::PageAllocator*) [node]
 5: 0xc53c23  [node]
 6: 0xc545b4 node::Start(int, char**) [node]
 7: 0x7f6da641bd90  [/lib/x86_64-linux-gnu/libc.so.6]
 8: 0x7f6da641be40 __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]
 9: 0xbb0c7e _start [node]
Aborted (core dumped)

Docker is pretty old: Docker version 19.03.10, build 9424aeaee9, but sadly I am unable to upgrade it at the moment. I thought that the latest node version should work?

[kamalkech]

i have the same, what is the correct solution ?

[LeoK80]

i have the same, what is the correct solution ?

Try upgrade to latest container runtime (docker, containerd, etc.) to latest. If nothing newer is picked up from your package manager, consider upgrading manually.

[kode54]

Experiencing this with nodejs 22.0.0-1 from official Arch packages, on x86_64, as of kernel 6.9.1. (Also experienced it with -rc7 before that, so I downgraded to 6.8.9 at the time.)

[getong]

archlinux 6.9.1, nodejs 22.0.0-1, same error

[colddegree]

Same problem

➜  ~ uname -a
Linux tuf 6.9.1-arch1-1 #1 SMP PREEMPT_DYNAMIC Fri, 17 May 2024 16:56:38 +0000 x86_64 GNU/Linux
➜  ~ pacman -Q | grep node        
node-gyp 10.1.0-2
nodejs 22.2.0-1
nodejs-nopt 7.2.0-1

[daniluk4000]

Btw we still use bullseye, works pretty good

[Vortetty]

not to bring up an old issue again but this appears to be a reoccurring bug. reproducing it consistently with kernel 6.9.1-arch1 and node-gyp 10.1.0-2 nodejs 22.2.0-1 nodejs-nopt 7.2.0-1 packages on Crystal Linux (basically arch)

it is fixed with the UV_USE_IORING=0 tag, but i'd rather not have to use that if a fix can be found for newer kernels as well

given that that does fix it, it may also podentially be an issue with libuv-1.48.0-2

[mcollina]

cc @santigimeno can you take another look?

[richardlau]

it is fixed with the UV_USE_IORING=0 tag, but i'd rather not have to use that if a fix can be found for newer kernels as well

This is supposed to be the default for Node.js (since the February security releases).

https://nodejs.org/docs/latest-v22.x/api/cli.html#uv_use_io_uringvalue

io_uring is disabled by default due to security concerns.

[santigimeno]

A patch has just been sent to the kernel fixing this: https://lore.kernel.org/io-uring/40c7404a-f4ce-4a7d-86f3-313a9e9ee113@kernel.dk/T/#u

It should land in stable shortly: https://lore.kernel.org/io-uring/d4cadead-369d-4d94-ac85-58034d75c6c7@kernel.dk/T/#t

[santigimeno]

This is supposed to be the default for Node.js (since the February security releases).

https://nodejs.org/docs/latest-v22.x/api/cli.html#uv_use_io_uringvalue

I just tested with 22.2.0 installed from nvm and as documented, io_uring is disabled there. Maybe is there a problem in the arch linux package?

[gengjiawen]

If you run into this on ubuntu 16 or 18, my fix is use ubuntu >= 20.04. The issue actually comes from docker for me.

[kode54]

This is supposed to be the default for Node.js (since the February security releases). https://nodejs.org/docs/latest-v22.x/api/cli.html#uv_use_io_uringvalue

I just tested with 22.2.0 installed from nvm and as documented, io_uring is disabled there. Maybe is there a problem in the arch linux package?

How is arch supposed to be disabling io_uring? It configures nodejs to use the system libuv, and builds its libuv with the default options.

[santigimeno]

How is arch supposed to be disabling io_uring? It configures nodejs to use the system libuv, and builds its libuv with the default options.

That's likely the problem. Due to the security reasons mentioned above node.js patched libuv to disable io_uring in the following commits: https://github.com/nodejs/node/commit/42e659cb9d9425f76dbe9b57a437005508c0933d and https://github.com/nodejs/node/commit/6d14352c51974f0ba1a11e9e4889e61dae9da1f4. Maybe the arch packaging hasn't taken that into account?

[mcollina]

Thanks @santigimeno for the help debugging this. This issue can indeed be closed.

[felixonmars]

How is arch supposed to be disabling io_uring? It configures nodejs to use the system libuv, and builds its libuv with the default options.

That's likely the problem. Due to the security reasons mentioned above node.js patched libuv to disable io_uring in the following commits: 42e659c and 6d14352. Maybe the arch packaging hasn't taken that into account?

I am the Arch packager and indeed I have missed this change.

Opened https://github.com/libuv/libuv/issues/4416 to see if there is a better way forward.