Memory leak with NestJS and NestJS-graphql

[rskaar]

What version of Bun is running?

1.1.21

What platform is your computer?

Darwin 23.5.0 arm64 arm / Docker with base image oven/bun:1.1.21

What steps can reproduce the bug?

Minimal reproduction repo, with code and documented steps to reproduce: https://github.com/rskaar/bun-nestjs-graphql-mem-issues

What is the expected behavior?

The problem occurs when the NestJS app receives requests to any endpoint, as long as the NestJS project has the NestJS-graphQL module imported. For each request, the memory consumption goes up. After 100-300.000 requests, we're at over 1 gb of memory consumption from an initial <100mb.

The same app/code running with NodeJS does not have the same memory leak.

Removing (commenting out) the imported NestJS-GraphQL module gives no memory issues while running.

Note that the requests leading to the memory issues does not need to hit the GraphQL-endpoint. Any traffic to any endpoint in the NestJS app seems to generate a memory leak. I suspect that the NestJS-GraphQL (or ApolloServer) module adds some kind of middleware that each request goes through.

What do you see instead?

Memory usage goes up for each request and is not garbage collected.

Additional information

We use Bun in production, and this is a major issue for us - leading to our docker crashing multiple times per day due to memory usage.

[Jarred-Sumner]

If these are incoming requests with a request body, can you run bun upgrade --canary and see if it’s better?

[rskaar]

Tried upgrading to canary, but the issues persist. The requests used for testing are just GET http://locahost:3000/ without any body, special headers or similar.

[billywhizz]

i've taken an initial look at this and can repro on ubuntu 22.04 following the excellent guide from @rskaar! :pray:

i can see JSC heap usage growing consistently and not shrinking despite explicit calls to flush any garbage. i will try to figure out what is causing it.

this is code i am using to capture the heap stats.

import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';
import {
  gcAndSweep,
  heapSize,
  heapStats,
} from "bun:jsc";

const AD = '\u001b[0m' // ANSI Default
const AG = '\u001b[32m' // ANSI Green
const AY = '\u001b[33m' // ANSI Yellow

function to_size_string (bytes) {
  if (bytes < 1000) {
    return `${bytes.toFixed(2).padStart(8, ' ')} ${AY} Bps${AD}`
  } else if (bytes < 1000 * 1000) {
    return `${(Math.floor((bytes / 1000) * 100) / 100).toFixed(2).padStart(8, ' ')} ${AY}KBps${AD}`
  } else if (bytes < 1000 * 1000 * 1000) {
    return `${(Math.floor((bytes / (1000 * 1000)) * 100) / 100).toFixed(2).padStart(8, ' ')} ${AY}MBps${AD}`
  }
  return `${(Math.floor((bytes / (1000 * 1000 * 1000)) * 100) / 100).toFixed(2).padStart(8, ' ')} ${AY}GBps${AD}`
}

async function bootstrap() {
  const app = await NestFactory.create(AppModule);
  await app.listen(3000);
}
bootstrap();

setInterval(() => {
  gcAndSweep()
  Bun.gc(true)
  const { rss } = process.memoryUsage()
  const heap_size = heapSize()
  const stats = heapStats()
  console.log(`${AG}rss${AD} ${to_size_string(rss)} ${AG}heap${AD} ${to_size_string(heap_size)}, ${AG}stats.heap${AD} ${to_size_string(stats.heapSize)} ${AG}stats.count${AD} ${stats.objectCount}`)
}, 1000)

[billywhizz]

some further info from triage. it looks like we have a bunch of objects that get created on every request and are not being marked as dead for GC in Bun. Using a slightly modified script to keep track of JS Values on the heap we can see

rss     1.26 GBps heap   888.47 MBps, stats.heap   888.47 MBps stats.count 11123093
{
  "Arguments": 4399,
  "Array": 57187,
  "BufferList": 8798,
  "Function": 61586,
  "FunctionCodeBlock": 3,
  "FunctionExecutable": 1,
  "Headers": 4399,
  "JSLexicalEnvironment": 21995,
  "Object": 74783,
  "Promise": 4399,
  "PropertyTable": 21995,
  "Request": 4399,
  "Response": 4399,
  "Structure": 61571,
  "StructureRareData": 21994,
  "SymbolTable": 2,
  "string": 13196
}

Those counts are new JS Values created since the previous tick of the timer (one second). Throughput/RPS of the server also gradually decreases as the heap size grows.

So, we get one Request object and a bunch of other objects hanging off it for every http request and these never get GC'd.

we can also see the count of objects on the heap steadily increasing as we load the server.

rss     1.16 GBps heap   811.35 MBps, stats.heap   811.35 MBps stats.count 10197646
{
  "Arguments": 4544,
  "Array": 59071,
  "BufferList": 9088,
  "Function": 63615,
  "Generator": 1,
  "Headers": 4544,
  "JSLexicalEnvironment": 22718,
  "Object": 77247,
  "Promise": 4543,
  "PropertyTable": 22720,
  "Request": 4544,
  "Response": 4544,
  "Structure": 63616,
  "StructureRareData": 22720,
  "string": 13490
}
rss     1.22 GBps heap   852.09 MBps, stats.heap   852.09 MBps stats.count 10661221
{
  "Arguments": 4415,
  "Array": 57395,
  "BufferList": 8830,
  "Function": 61810,
  "Headers": 4415,
  "JSLexicalEnvironment": 22075,
  "Object": 75055,
  "Promise": 4415,
  "PropertyTable": 22075,
  "Request": 4415,
  "Response": 4415,
  "Structure": 61810,
  "StructureRareData": 22075,
  "string": 13245
}
rss     1.26 GBps heap   888.47 MBps, stats.heap   888.47 MBps stats.count 11123093
{
  "Arguments": 4399,
  "Array": 57187,
  "BufferList": 8798,
  "Function": 61586,
  "FunctionCodeBlock": 3,
  "FunctionExecutable": 1,
  "Headers": 4399,
  "JSLexicalEnvironment": 21995,
  "Object": 74783,
  "Promise": 4399,
  "PropertyTable": 21995,
  "Request": 4399,
  "Response": 4399,
  "Structure": 61571,
  "StructureRareData": 21994,
  "SymbolTable": 2,
  "string": 13196
}
rss     1.30 GBps heap   924.51 MBps, stats.heap   924.51 MBps stats.count 11580083
{
  "Arguments": 4352,
  "Array": 56576,
  "BufferList": 8704,
  "Function": 60928,
  "FunctionCodeBlock": 2,
  "FunctionExecutable": 1,
  "Headers": 4352,
  "JSLexicalEnvironment": 21760,
  "Object": 73984,
  "Promise": 4352,
  "PropertyTable": 21760,
  "Request": 4352,
  "Response": 4352,
  "Structure": 60944,
  "StructureRareData": 21760,
  "SymbolTable": 2,
  "string": 13065
}

I also verified node.js does not show this behaviour even against the bundled/compiled JS generated by bun. So, it's definitely a memory leak in Bun/JSC internals. Hopefully won't be hard to fix.

[rskaar]

Just a note, the issue is still there in Bun 1.1.26