[Bug]: `@redwoodjs/testing/config/jest/api` causes a memory leak on api side tests

[jmackie]

What's not working?

At Nous our Redwood test suite has become incredibly resource hungry, to the point that we can't actually run the entire suite on our local machines. We can just about run it on CI by splitting the work between multiple GHA jobs.

After some investigation it's clear that we have a huge memory leak. And one of the contributors seems to be @redwoodjs/testing/config/jest/api, which leaks memory even with the most trivial test suite.

Apologies if this is something you're already aware of - I searched for similar issues but no luck 🙈

Otherwise our experience with Redwood remains very positive, thank you team for all your hard work 🙏

How do we reproduce the bug?

https://github.com/project-nous/redwood-jest-memory-leak-repro

What's your environment? (If it applies)

No response

Are you interested in working on this?

• [X] I'm interested in working on this

[zygopleural]

From my own debugging, with the standard repro I get

 PASS   api  api/src/7.test.ts (158 MB heap size)
 PASS   api  api/src/14.test.ts (204 MB heap size)
 PASS   api  api/src/6.test.ts (286 MB heap size)
 PASS   api  api/src/15.test.ts (364 MB heap size)
 PASS   api  api/src/17.test.ts (444 MB heap size)
 PASS   api  api/src/4.test.ts (523 MB heap size)
 PASS   api  api/src/20.test.ts (354 MB heap size)
 PASS   api  api/src/16.test.ts (431 MB heap size)
 PASS   api  api/src/5.test.ts (510 MB heap size)
 PASS   api  api/src/13.test.ts (591 MB heap size)
 PASS   api  api/src/0.test.ts (670 MB heap size)
 PASS   api  api/src/9.test.ts (508 MB heap size)
 PASS   api  api/src/8.test.ts (589 MB heap size)
 PASS   api  api/src/12.test.ts (668 MB heap size)
 PASS   api  api/src/1.test.ts (747 MB heap size)
 PASS   api  api/src/19.test.ts (630 MB heap size)
 PASS   api  api/src/3.test.ts (703 MB heap size)
 PASS   api  api/src/10.test.ts (781 MB heap size)
 PASS   api  api/src/2.test.ts (862 MB heap size)
 PASS   api  api/src/11.test.ts (940 MB heap size)
 PASS   api  api/src/18.test.ts (709 MB heap size)

Test Suites: 21 passed, 21 total
Tests:       21 passed, 21 total
Snapshots:   0 total
Time:        12.48 s

If I comment everything out in node_modules/@redwoodjs/testing/config/jest/api/jest.setup.js, I get:

 PASS   api  api/src/7.test.ts (75 MB heap size)
 PASS   api  api/src/18.test.ts (84 MB heap size)
 PASS   api  api/src/20.test.ts (83 MB heap size)
 PASS   api  api/src/19.test.ts (91 MB heap size)
 PASS   api  api/src/2.test.ts (92 MB heap size)
 PASS   api  api/src/15.test.ts (85 MB heap size)
 PASS   api  api/src/9.test.ts (92 MB heap size)
 PASS   api  api/src/14.test.ts (92 MB heap size)
 PASS   api  api/src/13.test.ts (100 MB heap size)
 PASS   api  api/src/12.test.ts (90 MB heap size)
 PASS   api  api/src/3.test.ts (98 MB heap size)
 PASS   api  api/src/11.test.ts (99 MB heap size)
 PASS   api  api/src/1.test.ts (106 MB heap size)
 PASS   api  api/src/10.test.ts (108 MB heap size)
 PASS   api  api/src/4.test.ts (115 MB heap size)
 PASS   api  api/src/0.test.ts (98 MB heap size)
 PASS   api  api/src/16.test.ts (106 MB heap size)
 PASS   api  api/src/8.test.ts (105 MB heap size)
 PASS   api  api/src/17.test.ts (113 MB heap size)
 PASS   api  api/src/6.test.ts (115 MB heap size)
 PASS   api  api/src/5.test.ts (123 MB heap size)

Test Suites: 21 passed, 21 total
Tests:       21 passed, 21 total
Snapshots:   0 total
Time:        1.089 s, estimated 13 s

If I just comment out the beforeAll / afterAll / afterEach hooks in node_modules/@redwoodjs/testing/config/jest/api/jest.setup.js, I get:

 PASS   api  api/src/7.test.ts (131 MB heap size)
 PASS   api  api/src/1.test.ts (196 MB heap size)
 PASS   api  api/src/19.test.ts (261 MB heap size)
 PASS   api  api/src/6.test.ts (321 MB heap size)
 PASS   api  api/src/4.test.ts (380 MB heap size)
 PASS   api  api/src/3.test.ts (446 MB heap size)
 PASS   api  api/src/8.test.ts (297 MB heap size)
 PASS   api  api/src/9.test.ts (356 MB heap size)
 PASS   api  api/src/13.test.ts (421 MB heap size)
 PASS   api  api/src/12.test.ts (481 MB heap size)
 PASS   api  api/src/16.test.ts (547 MB heap size)
 PASS   api  api/src/5.test.ts (410 MB heap size)
 PASS   api  api/src/20.test.ts (471 MB heap size)
 PASS   api  api/src/14.test.ts (535 MB heap size)
 PASS   api  api/src/10.test.ts (594 MB heap size)
 PASS   api  api/src/17.test.ts (659 MB heap size)
 PASS   api  api/src/15.test.ts (520 MB heap size)
 PASS   api  api/src/18.test.ts (583 MB heap size)
 PASS   api  api/src/11.test.ts (643 MB heap size)
 PASS   api  api/src/2.test.ts (707 MB heap size)
 PASS   api  api/src/0.test.ts (767 MB heap size)

Test Suites: 21 passed, 21 total
Tests:       21 passed, 21 total
Snapshots:   0 total
Time:        9.762 s, estimated 11 s

If I keep the beforeAll / afterAll / afterEach hooks in node_modules/@redwoodjs/testing/config/jest/api/jest.setup.js commented out and update some of the imports, namely:

-const { getConfig, getDMMF } = require('@prisma/sdk')
+const { getConfig, getDMMF } = require('@prisma/sdk/dist/engine-commands')

-const { setContext } = require('@redwoodjs/graphql-server')
+const { setContext } = require('@redwoodjs/graphql-server/dist/globalContext')
 const { getPaths } = require('@redwoodjs/internal/dist/paths')
-const { defineScenario } = require('@redwoodjs/testing/dist/api')
+
+const { defineScenario } = require('../../../dist/api/scenario')

I get:

 PASS   api  api/src/7.test.ts (94 MB heap size)
 PASS   api  api/src/8.test.ts (113 MB heap size)
 PASS   api  api/src/19.test.ts (114 MB heap size)
 PASS   api  api/src/16.test.ts (134 MB heap size)
 PASS   api  api/src/5.test.ts (153 MB heap size)
 PASS   api  api/src/12.test.ts (137 MB heap size)
 PASS   api  api/src/2.test.ts (157 MB heap size)
 PASS   api  api/src/4.test.ts (174 MB heap size)
 PASS   api  api/src/10.test.ts (192 MB heap size)
 PASS   api  api/src/14.test.ts (210 MB heap size)
 PASS   api  api/src/6.test.ts (228 MB heap size)
 PASS   api  api/src/9.test.ts (246 MB heap size)
 PASS   api  api/src/1.test.ts (264 MB heap size)
 PASS   api  api/src/15.test.ts (282 MB heap size)
 PASS   api  api/src/11.test.ts (300 MB heap size)
 PASS   api  api/src/17.test.ts (318 MB heap size)
 PASS   api  api/src/20.test.ts (336 MB heap size)
 PASS   api  api/src/13.test.ts (354 MB heap size)
 PASS   api  api/src/18.test.ts (372 MB heap size)
 PASS   api  api/src/0.test.ts (390 MB heap size)
 PASS   api  api/src/3.test.ts (408 MB heap size)

Test Suites: 21 passed, 21 total
Tests:       21 passed, 21 total
Snapshots:   0 total
Time:        3.174 s, estimated 4 s

i.e. heap is not increasing at such a fast pace and the tests are running significantly faster due to smaller import sizes (i.e. not importing the whole of the Prisma SDK.

[jmackie]

Not sure if it's related, but it's also slightly alarming that this trivial test suite is taking >10s 😅

[zygopleural]

If I keep the beforeAll / afterAll / afterEach hooks in node_modules/@redwoodjs/testing/config/jest/api/jest.setup.js commented out and update some of the imports, namely:

-const { getConfig, getDMMF } = require('@prisma/sdk')
+const { getConfig, getDMMF } = require('@prisma/sdk/dist/engine-commands')

-const { setContext } = require('@redwoodjs/graphql-server')
+const { setContext } = require('@redwoodjs/graphql-server/dist/globalContext')
 const { getPaths } = require('@redwoodjs/internal/dist/paths')
-const { defineScenario } = require('@redwoodjs/testing/dist/api')
+
+const { defineScenario } = require('../../../dist/api/scenario')

I get:

 PASS   api  api/src/7.test.ts (94 MB heap size)
 PASS   api  api/src/8.test.ts (113 MB heap size)
 PASS   api  api/src/19.test.ts (114 MB heap size)
 PASS   api  api/src/16.test.ts (134 MB heap size)
 PASS   api  api/src/5.test.ts (153 MB heap size)
 PASS   api  api/src/12.test.ts (137 MB heap size)
 PASS   api  api/src/2.test.ts (157 MB heap size)
 PASS   api  api/src/4.test.ts (174 MB heap size)
 PASS   api  api/src/10.test.ts (192 MB heap size)
 PASS   api  api/src/14.test.ts (210 MB heap size)
 PASS   api  api/src/6.test.ts (228 MB heap size)
 PASS   api  api/src/9.test.ts (246 MB heap size)
 PASS   api  api/src/1.test.ts (264 MB heap size)
 PASS   api  api/src/15.test.ts (282 MB heap size)
 PASS   api  api/src/11.test.ts (300 MB heap size)
 PASS   api  api/src/17.test.ts (318 MB heap size)
 PASS   api  api/src/20.test.ts (336 MB heap size)
 PASS   api  api/src/13.test.ts (354 MB heap size)
 PASS   api  api/src/18.test.ts (372 MB heap size)
 PASS   api  api/src/0.test.ts (390 MB heap size)
 PASS   api  api/src/3.test.ts (408 MB heap size)

Test Suites: 21 passed, 21 total
Tests:       21 passed, 21 total
Snapshots:   0 total
Time:        3.174 s, estimated 4 s

i.e. heap is not increasing at such a fast pace and the tests are running significantly faster due to smaller import sizes (i.e. not importing the whole of the Prisma SDK.

FYI, if I put the beforeAll / afterAll / afterEach hooks in node_modules/@redwoodjs/testing/config/jest/api/jest.setup.js back in but keep the import updates, I get:

 PASS   api  api/src/7.test.ts (114 MB heap size)
 PASS   api  api/src/8.test.ts (161 MB heap size)
 PASS   api  api/src/19.test.ts (196 MB heap size)
 PASS   api  api/src/12.test.ts (240 MB heap size)
 PASS   api  api/src/9.test.ts (287 MB heap size)
 PASS   api  api/src/18.test.ts (333 MB heap size)
 PASS   api  api/src/16.test.ts (380 MB heap size)
 PASS   api  api/src/14.test.ts (426 MB heap size)
 PASS   api  api/src/4.test.ts (472 MB heap size)
 PASS   api  api/src/5.test.ts (307 MB heap size)
 PASS   api  api/src/10.test.ts (353 MB heap size)
 PASS   api  api/src/20.test.ts (399 MB heap size)
 PASS   api  api/src/0.test.ts (445 MB heap size)
 PASS   api  api/src/1.test.ts (492 MB heap size)
 PASS   api  api/src/3.test.ts (539 MB heap size)
 PASS   api  api/src/2.test.ts (585 MB heap size)
 PASS   api  api/src/15.test.ts (631 MB heap size)
 PASS   api  api/src/13.test.ts (472 MB heap size)
 PASS   api  api/src/6.test.ts (519 MB heap size)
 PASS   api  api/src/11.test.ts (565 MB heap size)
 PASS   api  api/src/17.test.ts (611 MB heap size)

Test Suites: 21 passed, 21 total
Tests:       21 passed, 21 total
Snapshots:   0 total
Time:        5.917 s

i.e. heap still out of control, but tests are significantly quicker

[dac09]

Hello both - thank you for reporting this and the reproduction.

I just merged in a few changes (mainly targetting the web side) in https://github.com/redwoodjs/redwood/pull/6281 - would you like to try this out in your real project to see if it improves things for you?

yarn rw upgrade -t canary - we're very close to a v3 release and the canary isn't quite ready for prime time yet, but would be a great if you could give it a try.

I'll try it in your reproduction in the mean time.

[realStandal]

Possibly related to #4360

[dac09]

Hey so spent some time trying to improve speed/memory here - but I want to point out something important - the reproduction doesn't actually have any scenarios, or real usecases.

It's a tradeoff between being able to write scenarios, and seed the db without more boilerplate in each test vs keeping jest as vanilla as possible.

[jtoar]

Adding this one to the current cycle just so we keep seeing it and thinking about next steps.

[jan-stehlik]

hey @dac09 @jtoar is this being looked into please? memory leak is killing our apps test and our CI bills. We have to introduce excessive (and expensive) parallelism in order to have small enough test suites, so that they don't eat away all the memory. This is a major issue for us and causes so much pain that we don't really consider redwood test helpers production ready and are considering moving away from it unless this gets resolved

[jtoar]

@jan-stehlik it's still on our radar, but progress is sporadic right now. Preparing for v5 (react 18) is taking most of our attention; at first it seemed like it would take a long time but things are looking optimistic, so our attention may return to this fairly soon.

We've heard many complaints about the speed but not as many about the memory. I know those are related to some extent. Unless you've already connected with one of us, maybe it's worth connecting to see what you're doing in your project and if we can offer any tips to alleviate things for the time being?