search

webpack / enhanced-resolve

Offers an async require.resolve function. It's highly configurable.
MIT License
916 stars 186 forks source link

Reduce memory footprint of storing fs erros in cache #393

Open shYkiSto opened 10 months ago

shYkiSto commented 10 months ago

We are using enhanced-resolve together with jest test runner, and noticed substantial memory leaks using this combo. Further inspection revealed that original FS errors are retaining references to sandbox environments that jest creates to keep individual tests isolated from one another. These references occur in the internal representation of the Error object used for lazy-computation of the stack property later in the process (when user code happens to read from it for the first time).

Cloning the original error allows GC to collect that stack trace information and all retained objects that in turn helps to resolve the memory leak, and reduce the memory usage. One drawback to this is that the stack property would have to be computed eagerly, which comes at a cost of consuming more CPU. But since it's rarely required, and gets messed up due caching aspect of it anyway - a trade off has to be made to omit stack traces from the fs errors altogether.

linux-foundation-easycla[bot] commented 10 months ago

CLA Signed

The committers listed above are authorized under a signed CLA.

rtsao commented 8 months ago

Any feedback @alexander-akait @TheLarkInn ? This would be nice fix to get landed

Twipped commented 8 months ago

Is there a workaround for this that we can employ while we wait for this to be merged? We're seeing substantial memory problems in our jest suites that we've traced back to this exact issue.

alexander-akait commented 6 months ago

Sorry for delay, it sounds like the problem is with jest...

shYkiSto commented 6 months ago

Sorry for delay, it sounds like the problem is with jest...

Jest's approach to sandboxing definitely presents a challenge here. But at the same time there's not much that can be done there, as it's essentially the user code (enhanced-resolve) causing the leakage by holding references to the test environments that prevents them from being GC'd, as summarized in the PR description.

alexander-akait commented 4 months ago

@shYkiSto Sorry for the delay, can you measure the memory or demonstrate the leak (small reproduce example)?