search

cloudflare / workerd

The JavaScript / Wasm runtime that powers Cloudflare Workers
https://blog.cloudflare.com/workerd-open-source-workers-runtime/
Apache License 2.0
6.28k stars 304 forks source link

Debugging "allocation failed: JavaScript heap out of memory" #3120

Open godfat opened 1 week ago

godfat commented 1 week ago

We're using wrangler to run HTTP Router, and we hit into allocation failed: JavaScript heap out of memory error when we're running end-to-end tests against it.

The error looks like this:

✘ ERROR <--- Last few GCs --->

  [3455:0x106cffaa8000]  1294500 ms: Mark-Compact 1402.0 (1415.2) -> 1393.4 (1414.9) MB, pooled: 4 MB, 235.44 / 0.00 ms  (average mu = 0.908, current mu = 0.625) allocation failure; scavenge might not succeed
  [3455:0x106cffaa8000]  1297045 ms: Mark-Compact 1403.8 (1416.6) -> 1396.1 (1417.5) MB, pooled: 2 MB, 235.76 / 0.00 ms  (average mu = 0.908, current mu = 0.907) allocation failure; scavenge might not succeed

  <--- JS stacktrace --->

  workerd/jsg/setup.c++:38: fatal: V8 fatal error; location = Reached heap limit; message = : allocation failed: JavaScript heap out of memory

A full log can be downloaded at (will expire in 6 days) https://gitlab.com/gitlab-org/gitlab/-/jobs/8365282139/artifacts/external_file/gdk.gitlab-http-router.log

More investigation context can be found at: https://gitlab.com/gitlab-org/gitlab/-/issues/499983

So far we couldn't reproduce it locally, and when I tried to run memory profile on it locally, it only consumed about 2M, and as highest I can observe is about 10M, which was far from we see from the end-to-end tests: https://gitlab.com/gitlab-org/gitlab/-/issues/499983#note_2210491776

I wonder if you might have some tips for debugging this?

Let me know if you need any information that I could provide.

steve-mt commented 2 days ago

I've attached the expiring logs below so we don't loose them 🙇

gdk.gitlab-http-router.log.zip