Open MSDev201 opened 7 months ago
Hi @MSDev201 thanks for writing in!
This is a somewhat known issue although we're not entirely sure what exactly is causing it. So far we've mostly pinned it down to
Unfortunately, we don't have a solution for this yet which is also the case because we haven't received a minimal reproducible example yet. If you're able to provide one, we can debug this further.
If you have a highly customized build config by any chance, I suggest removing a couple of plugins to try to narrow it down. Let me know if this helps!
@Lms24 I could narrow it down a little more.
The Issue is happening here https://github.com/getsentry/sentry-javascript/blob/a7097d9ba2a74b2cb323da0ef22988a383782ffb/packages/sveltekit/src/vite/sourceMaps.ts#L169C7-L169C14
First of all, Im not sure if its correct to not await the flattening of the source maps. All processes are over when the flattening still is happening. My suggestion would be to change this line simply into:
for(const file of jsFiles) {
...
}
This doesnt solve the problem thoug but it makes more clear that the issue is not happening after the source maps are uploaded, but before.
To narrow it further down, I logged all files which where passed to sorcery. One of those files is a big 0.5mb js file with a 1mb source map. Sorcery appears to load also additional required source maps (possibly even circulating?) and I assume that this is where the out of memory error is caused. I havent looked into sorcery yet, so im not sure what options are there to prevent such issues.
First of all, Im not sure if its correct to not await the flattening of the source maps. All processes are over when the flattening still is happening.
oof you're completely right. This probably worked out by chance (more or less I'd guess), given the invocation of Sentry CLI to actually start uploading but it's definitely a bug. Thanks for flagging! I opened getsentry/sentry-javascript#10602 to fix
I havent looked into sorcery yet, so im not sure what options are there to prevent such issues.
Hmm this is a good point. Originally, I added sorcery to flatten the weird sourcemaps created by the Node Adapter (https://github.com/getsentry/sentry-javascript/pull/7811). Which adapter are you using @MSDev201? I wonder if we should simply scope flattening to the node adapter to avoid this for other ones.
Hmm this is a good point. Originally, I added sorcery to flatten the weird sourcemaps created by the Node Adapter (#7811). Which adapter are you using @MSDev201? I wonder if we should simply scope flattening to the node adapter to avoid this for other ones.
We are using @sveltejs/adapter-node version 2.0.0
So the issue in sorcery is that I get an infinite loop in the before mentioned source map. Im not very familiar with source maps and how they are supposed to look like, so either its an issue on my side because of a faulty site map or sorcery doesnt correctly handle circular references.
Quick simple example of my setup:
someFile.js
someCode();
//# sourceMappingURL=someFile.js.map
someFile.js.map
{
"sources": [
"someFile.js"
]
// ...
}
@Lms24 this issue was closed because of the fix
keyword in https://github.com/getsentry/sentry-javascript/pull/10602
@MSDev201 thanks a lot for investigating!
To me, this looks like there's already a problem in the generated source map but sorcery also shouldn't cause an infinite loop because of this. Were you able to pin this to a specific source file that causes the self-reference?
@Lms24 I was able to pin point which file caused it. Im not able to share it though. But I was able to reproduce generating such a file with source map so that it causes the infinite loop to happen.
The file which causes the problem for us, has a source map which is not generated by vite
, but was generated by another package with parcel
and is placed in the /static/
folder of the sveltekit project.
The following will cause the infinite loop (tested on two sveltekit projects):
static/test/test.js
console.log("Hello world!");
//# sourceMappingURL=test.js.map
static/test/test.js.map
{
"mappings": "ACAuB,QAAQ,GAAG,CAAC",
"sources": [
"<anon>",
"test.js"
],
"sourcesContent": [
"const $6c8bc724de647767$var$sayHello = ()=>console.log(\"Hello world!\");\n$6c8bc724de647767$var$sayHello();\n\n\n//# sourceMappingURL=test.js.map\n",
"const sayHello = () => console.log(\"Hello world!\");\nsayHello();\n"
],
"names": [
"console",
"log"
],
"version": 3,
"file": "test.js.map"
}
Running the build now will cause the infinite loop in sorcery, but because the file is so much smaller, it doesnt run into the 'out of memory' error but will cause the following error:
error during build:
Error: EMFILE: too many open files, open 'C:\Users\....\build\client\test\test.js.map'
Regarding the parcel build (I dont think its anything special):
We are running parcel build --target test
where the target is:
"test": {
"source": "test.js",
"outputFormat": "esmodule",
"distDir": "../static/test",
"optimize": true,
"includeNodeModules": []
}
and the source file test.js
looks like:
const sayHello = () => console.log("Hello world!");
sayHello();
Hey @MSDev201 thanks so much for investigating this further!
So this is a really painful problem because I'm afraid, to fix this on our end, we'll somehow need to check the source map content for such a circular reference which already smells brittle 😅 To confirm: In your case, this is the "first" source map, correct? "First", in the sense that the final bundle created by the node adapter points to the problematic source map, right?
For some context: The reason we need sorcery to flatten the source map is because the node adapter, in its own build step, will emit source maps that don't point to the original source code but to the code of the intermediate sveltekit build output. So if the circular reference actually happens in this intermediate source map, our plugin would effectively need to trace the source map chain itself, which sounds even more brittle.
Solution ideas:
I guess the most immediate workaround would be to expose an ignore option for flattening but this will affect the entire js file.
I have another idea how we could work around this but I'm not sure if this would work well either and the outcome isn't significantly better: Right now we call sorcery directly in the process of the plugin. We try/catch the calls to sorcery but obv this doesn't catch out of memory errors where the entire process crashes. We could explore invoking sorcery over its CLI in a separate process. This won't prevent the error but at least it won't crash the build and hence our plugin could even try to work around this
Ideally though, this can be fixed upstream in sorcery
. So it'd be best to open an issue in their repository. However, for this we need an isolated reproduction case, showing that it's really sorcery causing the error. I currently don't have the bandwidth for that. So if you could provide this it'd be greatly appreciated; otherwise I'm gonna backlog this and try to get to it after we shipped v8 of our SDKs.
Even more ideally, the Node adapter would 1. not have its own build process or 2. if really necessary, the emitted source maps would just point to the actual source code (i.e. no flattening needed). IIRC There was an issue about this in the sveltekit repo and someone attempted it, causing some errors, so they reverted it. I'm gonna have a look if this can be revisited.
UPDATE: found the issue in the sveltekit repo: https://github.com/sveltejs/kit/issues/10040
Is there an existing issue for this?
How do you use Sentry?
Self-hosted/on-premise
Which SDK are you using?
@sentry/sveltekit
SDK Version
7.100.1
Framework Version
Sveltekit 2.4.0
Link to Sentry event
No response
SDK Setup
vite.config.js
Steps to Reproduce
npx @sentry/wizard@latest -i sveltekit
and follow the promptsvite build
We are having some circular dependencies in a few of our dependencies, not sure if this could cause the issue.
Expected Result
Vite build should finish with a success code and dont consume all of my memory.
Actual Result
Sourcemap and Release are successfully published and it looks like everything has finished successfully but right before stopping, it consumes infinite amount of memory. I tested with
NODE_OPTIONS=--max-old-space-size=16384
but still get a 'out of memory' error.