Bun crashed after processing over a million results from an external API

Jelle-SamsonIT commented 1 week ago

How can we reproduce the crash?

I have no idea yet. I had my fresh code process a million search results from an external API that indexes public storage bucket

I cannot give a detailed crash report yet as I didn't encounter it again. There's 10k pages of HTTP requests happening here divided over 5 webworkers that process data from the external API and insert it into SQLite using many async/awaits.

Too complicated to trace, and i've only seen it happen once, but i wanted to get the crash report to you. There's probably some memory leaks in my code still too 😁

JavaScript/TypeScript code that reproduces the crash?

Relevant log output

No response

Stack Trace (bun.report)

Bun v1.1.5 (b257a30) on linux x86_64 [RunCommand]

Segmentation fault at address 0x00000000

ld-temp.o:0: WTF::Unicode::convertUTF16ToUTF8
1 unknown/js code

Jarred-Sumner commented 1 week ago

Are the Worker instances being closed at any point?

Jelle-SamsonIT commented 1 week ago

Negative. I added a worker.terminate() just to be sure and freed the callback. That may be where my memory leak came from: I'm setting worker[index].callback to be my callback to be executed when it's done:

Will try again this week and report back!

truongan07 commented 1 week ago

upgrade to v.1.1.7 was fixed leak memory in my case. You can try upgrade

oven-sh / bun