Open satdeveloping opened 4 months ago
@JonasBa could you help take a look? Thanks!
@satdeveloping I'm going to take a look at this, but I'm at a conference all day tomorrow so I will only be able to check this on Friday. Meanwhile, would you mind testing if other nodejs versions work or if this is isolated to the node20 runtime?
Ideally, it would be great if you can share a dockerfile that we can use to reproduce the crash so that I can try to reproduce the issue and attempt to extract more information from the crash
I have the same problem but with node:22.2.0-alpine
:
$ yarn install
yarn install v1.22.19
[1/4] Resolving packages...
[2/4] Fetching packages...
(node:91) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
[3/4] Linking dependencies...
warning " > @ovea-dev/eslint-plugin@1.14.52" has unmet peer dependency "@graphql-eslint/eslint-plugin@3.20.1".
warning " > @ovea-dev/eslint-plugin@1.14.52" has unmet peer dependency "@vue/compiler-sfc@3.4.27".
warning " > @ovea-dev/eslint-plugin@1.14.52" has unmet peer dependency "graphql@16.8.1".
warning Workspaces can only be enabled in private projects.
[4/4] Building fresh packages...
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
error /builds/ovea-dev/si-espace-client/ap/ru/ip/node_modules/@sentry/profiling-node: Command failed.
Exit signal: SIGSEGV
Command: node scripts/check-build.js
Arguments:
Directory: /builds/ovea-dev/si-espace-client/ap/ru/ip/node_modules/@sentry/profiling-node
Output:
@sentry/profiling-node: Precompiled binary found, attempting to load /builds/ovea-dev/si-espace-client/ap/ru/ip/node_modules/@sentry/profiling-node/lib/sentry_cpu_profiler-linux-x64-musl-127.node
@sentry/profiling-node: Precompiled binary found, skipping build from source.
But it works just fine with node:21.7.3-alpine
.
Ideally, it would be great if you can share a dockerfile that we can use to reproduce the crash so that I can try to reproduce the issue and attempt to extract more information from the crash
I mean, from the docker command I gave, a Dockerfile would look like
FROM node:20-alpine
WORKDIR /app
RUN npm init -y
RUN npm install @sentry/profiling-node
[+] Building 13.8s (8/8) FINISHED docker:desktop-linux
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 128B 0.0s
=> [internal] load metadata for docker.io/library/node:20-alpine 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [1/4] FROM docker.io/library/node:20-alpine@sha256:291e84d956f1aff38454bbd3da38941461ad569a185c20aa289f71f37ea08e23 0.9s
=> => resolve docker.io/library/node:20-alpine@sha256:291e84d956f1aff38454bbd3da38941461ad569a185c20aa289f71f37ea08e23 0.9s
=> [auth] library/node:pull token for registry-1.docker.io 0.0s
=> [2/4] WORKDIR /app 0.0s
=> [3/4] RUN npm init -y 0.8s
=> ERROR [4/4] RUN npm install @sentry/profiling-node 11.9s
------
> [4/4] RUN npm install @sentry/profiling-node:
11.80 npm ERR! path /app/node_modules/@sentry/profiling-node
11.81 npm ERR! command failed
11.81 npm ERR! signal SIGSEGV
11.81 npm ERR! command sh -c node scripts/check-build.js
11.81 npm ERR! @sentry/profiling-node: Precompiled binary found, attempting to load /app/node_modules/@sentry/profiling-node/lib/sentry_cpu_profiler-linux-arm64-musl-115.node
11.81 npm ERR! @sentry/profiling-node: Precompiled binary found, skipping build from source.
11.81
11.81 npm ERR! A complete log of this run can be found in: /root/.npm/_logs/2024-05-24T18_46_01_725Z-debug-0.log
------
Dockerfile:7
--------------------
5 | RUN npm init -y
6 |
7 | >>> RUN npm install @sentry/profiling-node
8 |
--------------------
ERROR: failed to solve: process "/bin/sh -c npm install @sentry/profiling-node" did not complete successfully: exit code: 1
I tested node:(18|20|22)-alpine and the issue was only present on node:20-alpine
We've temporarily downgraded to 7.110.1 which is the most recent 7.X version that works for us with node:20-alpine. Unfortunately we also receive the same error as listed above.
I'm going to investigate this today, sorry everyone. For now, it would help if anyone experiencing this could post the exact image they are using as well as a reproducible dockerfile.
@satdeveloping Just wanted to confirm if in your case, this also happens sporadically. I am attempting to reproduce this and it seems that it sometimes succeeds.
I will try and extract a coredump and see what is going on here.
I've attempted downgrading alpine to 3.18 and can no longer reproduce the crash (on any of the node versions)
@satdeveloping @maxkramer @mat813 Unless you require 3.18+, you should be able to downgrade to 3.18 and mitigate this temporarily while I figure out the root cause of the issue.
I just tried with node:22.2.0-alpine3.18 (and, to be complete, 3.19 and 3.20) and it SIGSEGV the same as the node:22.2.0-alpine image. Well, to be fair, it SIGSEGV most of the time, about once every 20-50 times, yarn install works.
I hit this problem trying to updating to sentry 8, then I saw the recent commit so I went ahead and installed profiling-node from github and it fixed my issue.
We'll get a release out within the next 1-2 days - thanks for your patience everyone (shoutout @JonasBa for the fix!)
It's still somewhat of a tentative fix though, I will need to extract a coredump to see exactly where things go wrong. I tried attaching the segfault handler, but the package doesnt support alpine, so I need to invest more time into it.
We released this as part of https://github.com/getsentry/sentry-javascript/releases/tag/8.8.0, would appreciate it if you can upgrade and give it a try 🙏
Using node:22.2.0-alpine, and after updating to 8.8.0, I still get :
[4/4] Building fresh packages...
error /builds/ovea-dev/si-espace-client/ap/ru/icinga/node_modules/@sentry/profiling-node: Command failed.
Exit signal: SIGSEGV
Command: node scripts/check-build.js
Arguments:
Directory: /builds/ovea-dev/si-espace-client/ap/ru/icinga/node_modules/@sentry/profiling-node
Output:
@sentry/profiling-node: Precompiled binary found, attempting to load /builds/ovea-dev/si-espace-client/ap/ru/icinga/node_modules/@sentry/profiling-node/lib/sentry_cpu_profiler-linux-x64-musl-127.node
@sentry/profiling-node: Precompiled binary found, skipping build from source.
Thanks for the quick turnaround.
I tested again today for node:20-alpine3.(18|19|20)
and the install fails with alpine 3.19 and 3.20. I can confirm it works as expected on 3.18.
The failure message is still the same.
@JonasBa to answer your earlier question, with node:20-alpine
and node:20-alpine3.(19|20)
it fails 100% of the time.
Yeah, I am unsure what changed here. I am going to trying to extract a coredump, but I suspect something in 3.19 and above changed and it's causing the prebuild binaries from 3.17 to segfault.
We are in a bit of an uncomfortable position if that is the case, as it means that building binaries on 3.19 and above will break the runtimes of people running on <3.18.
@satdeveloping there is one way to avoid this and it would mean that you would need to install the tooling and build the bindings yourself.
A dockerfile for something like that would then look like
# Install the toolchain
RUN apk add --no-cache build-base git g++ make curl python3
# Install the packages and ignore scripts, which does not check if the binary had been compiled
RUN npm i --ignore-scripts
# Build the binary from source
RUN (cd node_modules/@sentry/profiling-node && npm run build:bindings:configure && npm run build:bindings)
This would then build the binaries for the alpine version you are using and should fix the issue (unless it originates from some other change)
Node 22.3 was released, and with node:22.3.0-alpine, the crash no longer occurs. Looking at the changelog, I am unsure about which part was fixed... :(
Can confirm was getting the same issue when using docker (on my m1 mac) with image node:20-alpine
, using node:22.3.0-alpine3.19
fixes the issue.
Ok, that is very weird. I'm going to go over the nodejs changelog to try and see if what might have caused this.
Just adding I've run into this (on my M2 mac) with v8.9.2 on node20.14.0-alpine
. Temporarily swapping to node22.3.0-alpine
for a short-term workaround.
I would recommend you to upgrade to LTS or downgrade alpine. It seems that whatever the issue was, it has gotten fixed upstream somewhere in nodejs. cc @anonrig if you know of any recent fixes in nodejs that might have caused this?
I would recommend you to upgrade to LTS or downgrade alpine. It seems that whatever the issue was, it has gotten fixed upstream somewhere in nodejs. cc @anonrig if you know of any recent fixes in nodejs that might have caused this?
I'm confused - isn't Node 20 LTS? And Node removed alpine 3.18 from its published tags. So neither of these seem actionable unless I've misunderstood.
Sorry, I misspoke. I meant either upgrade to 22 or try downgrading alpine. I'm tempted to upgrade alpine from 3.17 for our build images, but I'm hesitating because it will likely break existing apps unless we make it a major bump (which is something I would not want to do right now)
Any progress here? Having the same issue
Hey @clemente-xyz, from our point of view no updates so far. Gonna tag @JonasBa (who's off today) to confirm.
It seems the problem is only with Node 20 and Alpine 3.19/3.20. Node 20 with Alpine 3.18 works as well as Node 22.
Hey everyone, first of all, I'm sorry about the wait, I haven't given this proper care and figure out the root cause. I'm going to see if I can figure out what exactly is going on here. In the meantime, if anyone managed to extract a coredump and share it, I would be very happy.
As a possible workaround, if some changes in alpine are indeed the issue, you should be able to temporarily build from source
apk add build-base git g++ make curl python3
npm install --ignore-scripts
(cd node_modules/@sentry/profiling-node && npm run build:bindings:configure && npm run build:bindings)
I'm going to try and extract a coredump to see whats going on
This is happening in node:22.8-alpine conditionally.
I've created a reproducible repository with error occuring github actions over here.
In case this could be helpful I've also added valgrind log too.
This is a coredump from arm64 node:22.8-alpine docker image v8-compile-cache-0.zip.
Thank you @asomethings, that helps a lot.
There seems to be something suspicious here, or maybe I'm going crazy but if I rebuild the package on alpine3.19 or alpine3.20 without changing the source, the error no longer appears and everything works as expected.
@JonasBa This seems more crazy but it seems like node:22.6-alpine3.20
image installs perfectly normally. I'm not quite sure but maybe my issue could be unrelated with issues other people are suffering from.
Yes, @asomethings, something is definitely weird here. Right now what I've done is updated the close to check for is_closing
vs is_active
which is what that method actually asserts on + heap allocated the timer so that we can clean it up after the handler is closed (which is the recommended way to do that anyway).
Is there an existing issue for this?
How do you use Sentry?
Sentry Saas (sentry.io)
Which SDK are you using?
@sentry/node
SDK Version
8.2.1
Framework Version
No response
Link to Sentry event
No response
SDK Setup
No response
Steps to Reproduce
Expected Result
The package installs
Actual Result
There is a segfault.
npm debug log