Closed kozharskyad closed 1 month ago
Are you able to reproduce this outside of alpine linux? Musl LibC is not supported right now in Bun (#918), and using glibc compat can cause strange issues like crashes when loading e.g. Prisma
Edit: I think it's unlikely that alpine is the cause - this stacktraace is entirely coming from JSC.
Yes, I be able reproduce crash outside MUSL environment. Before Docker+Alpine+MUSL I run application in single-binary variant on Debian GNU/Linux 11 (bullseye). To make single binary, I run this command:
bun build --compile --target=bun-linux-x64-baseline --outfile=harbor-scanner-events-processor index.js
OR
bun build --compile --target=bun-linux-x64 --outfile=harbor-scanner-events-processor index.js
Run build on macOS Sonoma 14.5 or on Debian 11 - not important. Same result.
I haven't been able to reproduce it from the example on either linux x64 or macOS arm64 or on a debug build of macOS arm64, but one idea: if you move the resolve
call to happen after the readableStreamToJSON() call, does it still occur? Or otherwise, if you await
the readableStreamToJSON
call. Another thought: if you switch from readableStreamToJSON
to instead use await req.json()
?
This two variants produces same error:
1)
const processRequest = req => new Promise(resolve =>
req.json()
.then(json => (resolve(), processRawEvent(json)))
.catch(() => resolve())
)
2)
const processRequest = req =>
req.json()
.then(json => processRawEvent(json))
I miss important thing in the description of the bug, but this thing in Makefile:
--platform=linux/amd64
Only intel/amd64 binary have this bug. On arm64 processors all OK.
Make some refactoring, middle Promise now absent, but error not gone. Also bad statistics:
RSS: 1.07GB | Peak: 119.92MB | Commit: 1.07GB | Faults: 295
1.07GB - it's not normal, if this metric shows us resident memory consumption, seems like memory leak.
UPD: If I change line 73 and replace .then(processRawEvent)
to .then(console.log)
, error is gone.
UPD: If I comment out line 47 .forEach(resource => processResource(resource))
, error is gone.
UPD: If I comment out line 22 createResource(resource)
, error is gone. Interesting. Seems the problem in internal SQLite engine.
UPD: If I comment return undefined
from createResource
function in db.js and completely comment out call of createResourceQuery.run({...})
, error is gone. Problem is in INSERT query.
UPD: Tryed to turn on strict: true
mode for database, run the code, got an error about absent url
parameter. Then removed all dollar signs in input query variables and got fatal error about segmentation from description of the bug.
UPD: Tryed to check theory about conflicting concurrent SQLite queries. After isResourceExists
and deleteResource
insert 3 seconds timeout before doing createResource
. But got same fatal crash after exactly 3 seconds. Also process.nextTick not helps. So changed name of the issue to "Crash on SQLite insert query".
@kozharskyad in about an hour (once CI finishes), can you try running
bunx bun-pr 12079
This script downloads the build artifact of Bun from a PR.
Can you try running in that version of Bun and see if the issue still happens?
I haven't been able to verify this, but here are two guesses I want to test: 1) Is this happening due to running a SQLite query that takes long enough to run where the GC attempts to free the SQLStatement before the function finishes executing? I don't think that would happen regardless, but I added some code to ensure that doesn't happen 2) Is there a missing exception check when rebinding values? I added an exception check in each call site to check.
Probably no success with pipeline - https://github.com/oven-sh/bun/actions/runs/9642668629 Also I got an error:
Searching GitHub for artifact bun-linux-x64 from PR #12079...
65 | requestCopy.url = requestCopy.url.replace(/\bclient_secret=\w+/g, "client_secret=[REDACTED]").replace(/\baccess_token=\w+/g, "access_token=[REDACTED]");
66 | this.request = requestCopy;
67 | Object.defineProperty(this, "code", {
68 | get() {
69 | logOnceCode(
70 | new import_deprecation.Deprecation(
^
Deprecation: [@octokit/request-error] `error.code` is deprecated, use `error.status`.
at get (/tmp/bunx-0-bun-pr@latest/node_modules/@octokit/request-error/dist-node/index.js:70:11)
120 | request: requestOptions
121 | });
122 | }
123 | if (status >= 400) {
124 | const data = await getResponseData(response);
125 | const error = new import_request_error.RequestError(toErrorMessage(data), status, {
^
HttpError: Not Found - https://docs.github.com/rest/pulls/pulls#get-a-pull-request
code: "404"
at /tmp/bunx-0-bun-pr@latest/node_modules/@octokit/request/dist-node/index.js:125:21
Bun v1.1.16 (Linux x64 baseline)
Download artifact from https://github.com/oven-sh/bun/pull/12111 success, but:
root@77e65db42f8f:/opt/app# ./bun --version
Illegal instruction
It's on macOS, force linux/amd64 platform. Probably artifact from pipeline is not baseline version.
UPD: Ok, found --baseline
:)
UPD: Got the same error:
Bun Canary v1.1.17-canary.1 (b0db40dc) Linux x64 (baseline)
Args: "./bun" "run" "index.js"
Features: jsc http_server
Builtins: "bun:main" "bun:sqlite" "node:path"
Elapsed: 1187ms | User: 484ms | Sys: 114ms
RSS: 1.07GB | Peak: 96.52MB | Commit: 1.07GB | Faults: 18
panic(main thread): Segmentation fault at address 0x80
oh no: Bun has crashed. This indicates a bug in Bun, not your code.
To send a redacted crash report to Bun's team,
please file a GitHub issue using the link below:
https://bun.report/1.1.17/Br2b0db40dAiggB2q+ymE+1mO_o7m+8Dyol+8D_6t+98D_4ry28Dy9w45EA2AgI
@kozharskyad can you run bunx bun-pr https://github.com/oven-sh/bun/pull/12246
and try it on that branch?
I'm wondering if it's a JavaScriptCore bug. Specifically, I'm wondering if this PR https://github.com/WebKit/WebKit/pull/30298 fixes it (which is included in that branch above)
Sorry, but no success :(
mbp-m3p:harbor-scanner-events-processor a.kozharsky$ make crash
/Library/Developer/CommandLineTools/usr/bin/make FLAGS="-d --name $(basename $PWD)-crash" run; \
while ! curl 127.0.0.1:8123 >/dev/null 2>&1; do sleep 1; done
docker build --platform=linux/amd64 -t $(basename $PWD) .
DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
BuildKit is currently disabled; enable it by removing the DOCKER_BUILDKIT=0
environment-variable.
Sending build context to Docker daemon 860.2kB
Step 1/13 : FROM oven/bun:1.1.17-alpine AS build
---> 67df01aab33f
Step 2/13 : WORKDIR /usr/src
---> Using cache
---> 923a2361f6d3
Step 3/13 : COPY package.json bun.lockb ./
---> Using cache
---> 097009c40a52
Step 4/13 : RUN GITHUB_TOKEN='XXX' bunx bun-pr https://github.com/oven-sh/bun/pull/12246 --baseline && mv /tmp/bun-node-*/bun-*-pr12246 /usr/local/bin/bun
---> [Warning] The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
---> Running in 91b071126c60
Resolving dependencies
Resolved, downloaded and extracted [72]
Saved lockfile
Searching GitHub for artifact bun-linux-x64-baseline from PR #12246...
Choosing artifact from run that started Jun 30, 2024, 6:36:14 AM
Downloading bun-linux-x64-baseline from PR #12246 - https://github.com/oven-sh/bun/actions/runs/9730027652
Downloaded to:
/tmp/bun-node-bb66bba1b/bun-01e423fc1a96a62bdd64a1807965a2a5bcf1ba34-pr12246
To run the downloaded executable, use any of the following following commands:
bun-01e423fc1a96a62bdd64a1807965a2a5bcf1ba34-pr12246
bun-12246
bun-latest
---> Removed intermediate container 91b071126c60
---> 4e90ae476ebd
Step 5/13 : RUN bun install
---> [Warning] The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
---> Running in 0169e859f4bf
bun install v1.1.18-canary.1 (9f9009b8)
+ typescript@5.4.5
+ @types/bun@1.1.4
+ react-dom@18.3.1
11 packages installed [8.98s]
---> Removed intermediate container 0169e859f4bf
---> 3d2ac2aaf025
Step 6/13 : COPY . .
---> 9b8aa47d10bf
Step 7/13 : RUN bun build --target=bun --sourcemap=inline --outdir=build index.js
---> [Warning] The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
---> Running in a1ff78040e50
index.js 1772.31 KB
fonts-3271f15256abe4c8.css 447.79 KB
style-76221e884f72d11f.css 0.98 KB
[107ms] bundle 22 modules
---> Removed intermediate container a1ff78040e50
---> 28c50e6d445a
Step 8/13 : FROM oven/bun:1.1.17-distroless
---> 938ab078cf4c
Step 9/13 : WORKDIR /opt/app
---> Using cache
---> c4e6f9f03332
Step 10/13 : COPY --from=build /usr/local/bin/bun /usr/local/bin/bun
---> 3f0f629e31e7
Step 11/13 : COPY --from=build /usr/src/build/ ./
---> 7a7b9dc0b434
Step 12/13 : ENTRYPOINT ["/usr/local/bin/bun", "run", "index.js"]
---> [Warning] The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
---> Running in 81d2da00e5f0
---> Removed intermediate container 81d2da00e5f0
---> ea9960ee66b8
Step 13/13 : CMD []
---> [Warning] The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
---> Running in 3282d6d44083
---> Removed intermediate container 3282d6d44083
---> e82ca956176e
Successfully built e82ca956176e
Successfully tagged harbor-scanner-events-processor:latest
docker run -itp8123:8123 --rm -d --name harbor-scanner-events-processor-crash $(basename $PWD)
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
2dd175bac70f99b3bb1e0b69a5a54ebedfd7de14d6f2ea2c728d109f96862d51
timeout 5s curl -svkXPOST -d'{"type":"SCANNING_COMPLETED","event_data":{"resources":[{"digest":"sha256:f302741af6fa1e2ebfead5a7828d953816e8e9eca773d812ca8a95637d414401","tag":"latest","resource_url":"some-image-ref:latest","scan_overview":{"application/vnd.security.vulnerability.report; version=1.1":{"report_id":"d96e643f-a886-4687-8803-e3a3a0ee8725","scan_status":"Success","severity":"Critical","duration":2,"summary":{"total":77,"fixable":77,"summary":{"Critical":6,"High":31,"Low":4,"Medium":36}},"start_time":"2024-06-19T08:59:32Z","end_time":"2024-06-19T08:59:34Z","scanner":{"name":"Trivy","vendor":"Aqua Security","version":"v0.42.0"},"complete_percent":100}}}],"repository":{"name":"dev/xxx/xxx-front","namespace":"library","repo_full_name":"library/dev/xxx/xxx-front","repo_type":"public"}}}' 127.0.0.1:8123/process || true
* Trying 127.0.0.1:8123...
* Connected to 127.0.0.1 (127.0.0.1) port 8123
> POST /process HTTP/1.1
> Host: 127.0.0.1:8123
> User-Agent: curl/8.8.0
> Accept: */*
> Content-Length: 774
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 774 bytes
docker logs $(basename $PWD)-crash
============================================================
Bun Canary v1.1.18-canary.1 (9f9009b8) Linux x64 (baseline)
Linux Kernel v6.6.31 | glibc v2.31
Args: "/usr/local/bin/bun" "run" "index.js"
Features: jsc http_server
Builtins: "bun:main" "bun:sqlite" "node:path"
Elapsed: 1224ms | User: 509ms | Sys: 93ms
RSS: 1.07GB | Peak: 126.59MB | Commit: 1.07GB | Faults: 0
panic(main thread): Segmentation fault at address 0x80
oh no: Bun has crashed. This indicates a bug in Bun, not your code.
To send a redacted crash report to Bun's team,
please file a GitHub issue using the link below:
https://bun.report/1.1.18/Br29f9009bAiggB27k5mE+1mO_ozhl9Dyggl9D_6l5k9D_+7s98Dy5xx6EA2AgI
docker rm -f $(basename $PWD)-crash
harbor-scanner-events-processor-crash
Unbelievable, but crush address always the same: 0x80
@Jarred-Sumner @cirospaciari I check crash on every release after 1.1.14, and now crash not reproducing on version 1.1.22 :) Thanks! Seems issue is resolved now.
How can we reproduce the crash?
0) prerequisites:
OR
Difference of this machines probably means machine architecture is not that important. On
oven/bun:1.1.13-alpine
crash not reproducible.1) Clone repository:
2) Change CWD to cloned repository:
3) Run crash test:
You can also open Makefile and see "crash" task with key commands:
Relevant log output
Stack Trace (bun.report)
Bun v1.1.16 (
bf7b327
) on linux x86_64_baseline [AutoCommand]Segmentation fault at address 0x00000080
ld-temp.o:0
:WTF::jscSignalHandler
??
ld-temp.o:0
:llint_op_call_varargs
ld-temp.o:0
:llint_op_call_ignore_result
ld-temp.o:0
:llint_op_call
ld-temp.o:0
:llint_call_javascript
ld-temp.o:0
:JSC::Interpreter::executeCall