Sveltekit - Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

MSDev201 commented 7 months ago

Is there an existing issue for this?

[X] I have checked for existing issues https://github.com/getsentry/sentry-javascript/issues
[X] I have reviewed the documentation https://docs.sentry.io/
[X] I am using the latest SDK release https://github.com/getsentry/sentry-javascript/releases

How do you use Sentry?

Self-hosted/on-premise

Which SDK are you using?

@sentry/sveltekit

SDK Version

7.100.1

Framework Version

Sveltekit 2.4.0

Link to Sentry event

No response

SDK Setup

vite.config.js

import { sveltekit } from "@sveltejs/kit/vite";
import { sentrySvelteKit } from "@sentry/sveltekit";

/** @type {import('vite').UserConfig} */
const config = {
    plugins: [
        sentrySvelteKit({
            autoUploadSourceMaps: true,
            sourceMapsUploadOptions: {
                cleanArtifacts: true,
                rewrite: false,
            },
        }),
        sveltekit(),
    ],
};
export default config;

Steps to Reproduce

Use npx @sentry/wizard@latest -i sveltekit and follow the prompts
Run vite build
Get heap out of memory error

We are having some circular dependencies in a few of our dependencies, not sure if this could cause the issue.

Expected Result

Vite build should finish with a success code and dont consume all of my memory.

Actual Result

Sourcemap and Release are successfully published and it looks like everything has finished successfully but right before stopping, it consumes infinite amount of memory. I tested with NODE_OPTIONS=--max-old-space-size=16384 but still get a 'out of memory' error.

getsentry/sentry-javascript#17 101.3 [***-vite-plugin] Info: Successfully uploaded source maps.
getsentry/sentry-javascript#17 102.5 [***-vite-plugin] Info: Successfully finalized release.
getsentry/sentry-javascript#17 128.4 
getsentry/sentry-javascript#17 128.4 <--- Last few GCs --->
getsentry/sentry-javascript#17 128.4 
getsentry/sentry-javascript#17 128.4 [41:0x5a2e330]   123431 ms: Scavenge (reduce) 2031.6 (2064.8) -> 2031.4 (2064.8) MB, 9.7 / 0.0 ms  (average mu = 0.259, current mu = 0.269) allocation failure; 
getsentry/sentry-javascript#17 128.4 [41:0x5a2e330]   123455 ms: Scavenge (reduce) 2034.7 (2068.1) -> 2034.7 (2068.3) MB, 20.5 / 0.0 ms  (average mu = 0.259, current mu = 0.269) allocation failure; 
getsentry/sentry-javascript#17 128.4 [41:0x5a2e330]   123905 ms: Scavenge (reduce) 2035.4 (2069.0) -> 2035.4 (2069.0) MB, 5.0 / 0.0 ms  (average mu = 0.259, current mu = 0.269) allocation failure; 
getsentry/sentry-javascript#17 128.4 
getsentry/sentry-javascript#17 128.4 
getsentry/sentry-javascript#17 128.4 <--- JS stacktrace --->
getsentry/sentry-javascript#17 128.4 
getsentry/sentry-javascript#17 128.4 FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
getsentry/sentry-javascript#17 128.4  1: 0xbdd160 node::Abort() [node]
getsentry/sentry-javascript#17 128.4  2: 0xaf089c  [node]
getsentry/sentry-javascript#17 128.4  3: 0xdcc1d0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
getsentry/sentry-javascript#17 128.4  4: 0xdcc586 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
getsentry/sentry-javascript#17 128.4  5: 0xfcb005  [node]
getsentry/sentry-javascript#17 128.4  6: 0xfcb5b6 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]
getsentry/sentry-javascript#17 128.4  7: 0xfdd8a6  [node]
getsentry/sentry-javascript#17 128.4  8: 0xfde4e5 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
getsentry/sentry-javascript#17 128.4  9: 0xfba49f v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
getsentry/sentry-javascript#17 128.4 10: 0xfbb4e7 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
getsentry/sentry-javascript#17 128.4 11: 0xf9ac4a v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
getsentry/sentry-javascript#17 128.4 12: 0x13ae67d v8::internal::Runtime_AllocateInOldGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
getsentry/sentry-javascript#17 128.4 13: 0x18343f9  [node]
getsentry/sentry-javascript#17 128.8 Aborted (core dumped)
getsentry/sentry-javascript#17 ERROR: process "/bin/sh -c npm run build" did not complete successfully: exit code: 134

Lms24 commented 7 months ago

Hi @MSDev201 thanks for writing in!

This is a somewhat known issue although we're not entirely sure what exactly is causing it. So far we've mostly pinned it down to

specific libraries. Are you using Firebase by any chance? There's a version of the library that caused these issues (https://github.com/getsentry/sentry-javascript/issues/9383)
a high number of large emitted bundles (https://github.com/getsentry/sentry-javascript-bundler-plugins/issues/512)

Unfortunately, we don't have a solution for this yet which is also the case because we haven't received a minimal reproducible example yet. If you're able to provide one, we can debug this further.

If you have a highly customized build config by any chance, I suggest removing a couple of plugins to try to narrow it down. Let me know if this helps!

MSDev201 commented 6 months ago

@Lms24 I could narrow it down a little more.

The Issue is happening here https://github.com/getsentry/sentry-javascript/blob/a7097d9ba2a74b2cb323da0ef22988a383782ffb/packages/sveltekit/src/vite/sourceMaps.ts#L169C7-L169C14

First of all, Im not sure if its correct to not await the flattening of the source maps. All processes are over when the flattening still is happening. My suggestion would be to change this line simply into:

for(const file of jsFiles) {
...
}

This doesnt solve the problem thoug but it makes more clear that the issue is not happening after the source maps are uploaded, but before.

To narrow it further down, I logged all files which where passed to sorcery. One of those files is a big 0.5mb js file with a 1mb source map. Sorcery appears to load also additional required source maps (possibly even circulating?) and I assume that this is where the out of memory error is caused. I havent looked into sorcery yet, so im not sure what options are there to prevent such issues.

Lms24 commented 6 months ago

First of all, Im not sure if its correct to not await the flattening of the source maps. All processes are over when the flattening still is happening.

oof you're completely right. This probably worked out by chance (more or less I'd guess), given the invocation of Sentry CLI to actually start uploading but it's definitely a bug. Thanks for flagging! I opened getsentry/sentry-javascript#10602 to fix

I havent looked into sorcery yet, so im not sure what options are there to prevent such issues.

Hmm this is a good point. Originally, I added sorcery to flatten the weird sourcemaps created by the Node Adapter (https://github.com/getsentry/sentry-javascript/pull/7811). Which adapter are you using @MSDev201? I wonder if we should simply scope flattening to the node adapter to avoid this for other ones.

MSDev201 commented 6 months ago

Hmm this is a good point. Originally, I added sorcery to flatten the weird sourcemaps created by the Node Adapter (#7811). Which adapter are you using @MSDev201? I wonder if we should simply scope flattening to the node adapter to avoid this for other ones.

We are using @sveltejs/adapter-node version 2.0.0

MSDev201 commented 6 months ago

So the issue in sorcery is that I get an infinite loop in the before mentioned source map. Im not very familiar with source maps and how they are supposed to look like, so either its an issue on my side because of a faulty site map or sorcery doesnt correctly handle circular references.

Quick simple example of my setup:

someFile.js

someCode();
//# sourceMappingURL=someFile.js.map

someFile.js.map

{
    "sources": [
        "someFile.js"
    ]
    // ...
}

@Lms24 this issue was closed because of the fix keyword in https://github.com/getsentry/sentry-javascript/pull/10602

Lms24 commented 6 months ago

@MSDev201 thanks a lot for investigating!

To me, this looks like there's already a problem in the generated source map but sorcery also shouldn't cause an infinite loop because of this. Were you able to pin this to a specific source file that causes the self-reference?

MSDev201 commented 6 months ago

@Lms24 I was able to pin point which file caused it. Im not able to share it though. But I was able to reproduce generating such a file with source map so that it causes the infinite loop to happen.

The file which causes the problem for us, has a source map which is not generated by vite, but was generated by another package with parcel and is placed in the /static/ folder of the sveltekit project.

The following will cause the infinite loop (tested on two sveltekit projects):

static/test/test.js

console.log("Hello world!");
//# sourceMappingURL=test.js.map

static/test/test.js.map

{
    "mappings": "ACAuB,QAAQ,GAAG,CAAC",
    "sources": [
        "<anon>",
        "test.js"
    ],
    "sourcesContent": [
        "const $6c8bc724de647767$var$sayHello = ()=>console.log(\"Hello world!\");\n$6c8bc724de647767$var$sayHello();\n\n\n//# sourceMappingURL=test.js.map\n",
        "const sayHello = () => console.log(\"Hello world!\");\nsayHello();\n"
    ],
    "names": [
        "console",
        "log"
    ],
    "version": 3,
    "file": "test.js.map"
}

Running the build now will cause the infinite loop in sorcery, but because the file is so much smaller, it doesnt run into the 'out of memory' error but will cause the following error:

error during build:
Error: EMFILE: too many open files, open 'C:\Users\....\build\client\test\test.js.map'

Regarding the parcel build (I dont think its anything special): We are running parcel build --target test where the target is:

"test": {
    "source": "test.js",
    "outputFormat": "esmodule",
    "distDir": "../static/test",
    "optimize": true,
    "includeNodeModules": []
}

and the source file test.js looks like:

const sayHello = () => console.log("Hello world!");
sayHello();

Lms24 commented 6 months ago

Hey @MSDev201 thanks so much for investigating this further!

So this is a really painful problem because I'm afraid, to fix this on our end, we'll somehow need to check the source map content for such a circular reference which already smells brittle 😅 To confirm: In your case, this is the "first" source map, correct? "First", in the sense that the final bundle created by the node adapter points to the problematic source map, right?

For some context: The reason we need sorcery to flatten the source map is because the node adapter, in its own build step, will emit source maps that don't point to the original source code but to the code of the intermediate sveltekit build output. So if the circular reference actually happens in this intermediate source map, our plugin would effectively need to trace the source map chain itself, which sounds even more brittle.

Solution ideas:

I guess the most immediate workaround would be to expose an ignore option for flattening but this will affect the entire js file.
I have another idea how we could work around this but I'm not sure if this would work well either and the outcome isn't significantly better: Right now we call sorcery directly in the process of the plugin. We try/catch the calls to sorcery but obv this doesn't catch out of memory errors where the entire process crashes. We could explore invoking sorcery over its CLI in a separate process. This won't prevent the error but at least it won't crash the build and hence our plugin could even try to work around this
Ideally though, this can be fixed upstream in sorcery. So it'd be best to open an issue in their repository. However, for this we need an isolated reproduction case, showing that it's really sorcery causing the error. I currently don't have the bandwidth for that. So if you could provide this it'd be greatly appreciated; otherwise I'm gonna backlog this and try to get to it after we shipped v8 of our SDKs.
Even more ideally, the Node adapter would 1. not have its own build process or 2. if really necessary, the emitted source maps would just point to the actual source code (i.e. no flattening needed). IIRC There was an issue about this in the sveltekit repo and someone attempted it, causing some errors, so they reverted it. I'm gonna have a look if this can be revisited.

UPDATE: found the issue in the sveltekit repo: https://github.com/sveltejs/kit/issues/10040

getsentry / sentry-javascript