panic(main thread): Segmentation fault at address 0x0

ImBIOS commented 2 weeks ago

How can we reproduce the crash?

(edited to content from https://github.com/oven-sh/bun/issues/14405)

I'm running a ai.google.dev gemini code inside a docker container (containerized app), and this happen while I'm doing some "smoke" test before I deploy this container to the "BIG MUSCLE" cloud. I don't the ai gen code is the cause because in this particular script I'm running I have this setup and several custom utils, I'm trying to gather some data for my research:


  System:
    OS: Linux 6.8 Ubuntu 24.04.1 LTS 24.04.1 LTS (Noble Numbat)
    CPU: (8) x64 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHz
    Memory: 4.85 GB / 15.41 GB
    Container: Yes
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 20.18.0 - ~/.proto/shims/node
    Yarn: 3.4.1 - ~/.proto/shims/yarn
    npm: 10.8.1 - ~/.proto/shims/npm
    bun: 1.1.29 - ~/.bun/bin/bun
    Watchman: 2024.10.07.00 - /home/linuxbrew/.linuxbrew/bin/watchman
  Languages:
    Bash: 5.2.21 - /usr/bin/bash
    Go: 1.23.1 - /home/imbios/.proto/shims/go
    Java: 11.0.24 - /home/imbios/.cache/coursier/arc/https/github.com/adoptium/temurin11-binaries/releases/download/jdk-11.0.24%252B8/OpenJDK11U-jdk_x64_linux_hotspot_11.0.24_8.tar.gz/jdk-11.0.24+8/bin/javac
    Perl: 5.38.2 - /usr/bin/perl
    Python3: 3.12.7 - /home/linuxbrew/.linuxbrew/bin/python3
    Rust: 1.80.0 - /home/imbios/.cargo/bin/rustc
  Browsers:
    Chrome: 129.0.6668.89
  npmPackages:
    @catdevnull/twitter-scraper: ^0.14.5 => 0.14.5 
    @commander-js/extra-typings: ^12.1.0 => 12.1.0 
    @google/generative-ai: ^0.21.0 => 0.21.0 
    @logtail/pino: ^0.5.0 => 0.5.0 
    @t3-oss/env-nextjs: ^0.11.1 => 0.11.1 
    @types/bun: ^1.1.10 => 1.1.10 
    @types/k6: ^0.54.1 => 0.54.1 
    @types/node: ^22.7.4 => 22.7.4 
    @types/user-agents: ^1.0.4 => 1.0.4 
    bun-bagel: ^1.1.0 => 1.1.0 
    canvas: ^3.0.0-rc2 => 3.0.0-rc2 
    crawlee: ^3.11.5 => 3.11.5 
    csv-string: ^4.1.1 => 4.1.1 
    dotenv-checker: ^1.1.7 => 1.1.7 
    escape-string-regexp: ^5.0.0 => 5.0.0 
    p-debounce: ^4.0.0 => 4.0.0 
    pino: ^9.4.0 => 9.4.0 
    pino-pretty: ^11.2.2 => 11.2.2 
    puppeteer: * => 23.5.0 
    puppeteer-cluster: ^0.24.0 => 0.24.0 
    puppeteer-extra: ^3.3.6 => 3.3.6 
    puppeteer-extra-plugin-stealth: ^2.11.2 => 2.11.2 
    puppeteer-screen-recorder: 3.0.5 => 3.0.5 
    sharp: ^0.33.5 => 0.33.5 
    typescript: ^5.6.2 => 5.6.2 
    user-agents: latest => 1.1.325 
    zod: ^3.23.8 => 3.23.8

It can only reproduced once after I tried several times.

Relevant log output

============================================================
Bun v1.1.29 (6d43b366) Linux x64 (baseline)
Linux Kernel v6.10.4 | glibc v2.31
CPU: sse42 popcnt avx avx2 avx512
Args: "bun" "index.ts" "-b" "false" "-l" "trace" "--serial" "--make-it-work"
Features: jsc Bun.stdin(2) bunfig dotenv(138) fetch(31) http_server spawn(367) tsconfig_paths tsconfig(17) WebSocket(9) workers_spawned(137) workers_terminated(124)
Builtins: "bun:main" "node:assert" "node:async_hooks" "node:buffer" "node:child_process" "node:constants" "node:crypto" "node:dns" "node:events" "node:fs" "node:fs/promises" "node:http" "node:https" "node:module" "node:net" "node:os" "node:path" "node:path/win32" "node:perf_hooks" "node:process" "node:readline" "node:stream" "node:stream/promises" "node:string_decoder" "node:timers/promises" "node:tls" "node:tty" "node:url" "node:util" "node:util/types" "node:vm" "node:zlib" "node:worker_threads" "node:punycode" "ws" "node-fetch"
Elapsed: 554713ms | User: 71391ms | Sys: 16658ms
RSS: 0.02ZB | Peak: 1.18GB | Commit: 0.02ZB | Faults: 36117

panic(main thread): Segmentation fault at address 0x0
oh no: Bun has crashed. This indicates a bug in Bun, not your code.

To send a redacted crash report to Bun's team,
please file a GitHub issue using the link below:

 https://bun.report/1.1.29/Ba16d43b36GqqihkLuzm08D+8mOi7zlB+h4lBug+lB2zz73D_gm2kzDgm2kzD+r68yDA2AA

Illegal instruction

Stack Trace (bun.report)

Bun v1.1.29 (6d43b36) on linux x86_64_baseline [AutoCommand]

Segmentation fault at address 0x00000000

napi.cpp:447: NAPIFunction::call
1 unknown/js code
ld-temp.o:0: llint_op_call_ignore_result
ld-temp.o:0: llint_op_call_ignore_result
ld-temp.o:0: llint_call_javascript

Features: workers_spawned, workers_terminated, jsc, Bun.stdin, bunfig, dotenv, fetch, http_server, spawn, tsconfig_paths, tsconfig, WebSocket

ImBIOS commented 2 weeks ago

Let me know if you need more info. But, I don't currently know how to make reproduce, it because all of repro repos I made can't even reproduce the panic even once. By god's will, I'll keep it updated if I make a proper repo which constantly makes this panic error.

190n commented 1 week ago

Does this crash occur with the same code as #14405, #14505, and #14239? Or are those all separate issues?

It'll be pretty hard for us to make progress without a consistent reproduction, sadly. You could try setting the environment variable BUN_GARBAGE_COLLECTOR_LEVEL=1 (or 2 if you're patient) and using the CLI flag --smol which can make GC-related crashes more consistent.

ImBIOS commented 1 week ago

If my memory serves me right, it occurs in the same codebase, but I can't disclose the whole code. I will try to pinpoint which part of the code makes this happen and make a minimum reproducible repository for you guys to debug and fix easily.

For a temporary solution, I'm gonna limit my crawlee crawling queue to below 1000 for each run. Because it only happens when I crawl tons of pages.

Thanks for the support team of bun.

ImBIOS commented 1 week ago

@190n Do you know what potentially caused this issue?

Just faced the same panic after implementing the GC suggestions (in production :v).

[04:41:54.592] HEALTHCHECK (index.ts/1682): ok
============================================================
Bun v1.1.30 (7996d06b) Linux x64 (baseline)
Linux Kernel v5.15.0 | glibc v2.31
CPU: sse42 popcnt avx avx2 avx512
Args: "bun" "--smol" "index.ts" "-b" "false" "-l" "trace" "--serial" "--make-it-work"
Features: Bun.stdin(2) WebSocket(10) bunfig dotenv(111) fetch(320) http_server jsc spawn(456) tsconfig(18) tsconfig_paths workers_spawned(110) workers_terminated(104)
Builtins: "bun:main" "node:assert" "node:async_hooks" "node:buffer" "node:child_process" "node:constants" "node:crypto" "node:dns" "node:events" "node:fs" "node:fs/promises" "node:http" "node:https" "node:module" "node:net" "node:os" "node:path" "node:path/win32" "node:perf_hooks" "node:process" "node:readline" "node:stream" "node:stream/promises" "node:string_decoder" "node:timers/promises" "node:tls" "node:tty" "node:url" "node:util" "node:util/types" "node:vm" "node:zlib" "node:worker_threads" "node:punycode" "ws" "node-fetch"
Elapsed: 627074ms | User: 228229ms | Sys: 67448ms
RSS: 0.02ZB | Peak: 17.38GB | Commit: 0.02ZB | Faults: 121

panic(main thread): Illegal instruction at address 0x3DB7314
oh no: Bun has crashed. This indicates a bug in Bun, not your code.

To send a redacted crash report to Bun's team,
please file a GitHub issue using the link below:

 (redacted)

./main.sh: line 13:  1682 Illegal instruction     (core dumped) bun --smol index.ts -b false -l trace --serial --make-it-work
error: script "start" exited with code 132

Does ZB mean Zetta Byte (1 ZB = 1 billion terabytes (TB)) ???

190n commented 1 week ago

Does ZB mean Zetta Byte (1 ZB = 1 billion terabytes (TB)) ???

Yes, that number is corrupted and doesn't reflect the actual memory consumption. The uint64_t with all 1 bits (0xFFFFFFFFFFFFFFFF) has a decimal value of 18,446,744,073,709,551,615, or approximately 0.2 zetta.

Could you share the output of the command find node_modules -name '*.node'? This will print a list of native addons used by your dependencies which can help us narrow this crash down. It seems like the issue is related to a native addon because the stack trace you shared ended with NAPIFunction::call -- what likely happened is that the runtime called a function implemented in native code by one of your dependencies, and due to a bug in Bun we passed it some incorrect data that made it crash.

oven-sh / bun