search

oven-sh / bun

Incredibly fast JavaScript runtime, bundler, test runner, and package manager – all in one
https://bun.sh
Other
72.25k stars 2.58k forks source link

Possible memory leak when used with MongoDB #12117

Open Akalanka47000 opened 1 month ago

Akalanka47000 commented 1 month ago

What version of Bun is running?

1.1.15

What platform is your computer?

Linux 5.10.214-202.855.amzn2.x86_64 x86_64 (Docker alpine image)

What steps can reproduce the bug?

I might be wrong here, but repeated calls to MongoDB through both mongoose and the driver skyrockets memory and CPU usage and doesn't seem to get reclaimed over time. After running out of options I ran the manual garbage collector at a recurring interval which reduced the rate at which memory is getting consumed but the issue still persists. The following drive link contains 3 heap snapshots taken from a running container at very short intervals :-

https://drive.google.com/file/d/1HG-68CdprR_nRwBkFInOOTBkS1VoneNA/view?usp=drive_link

Unfortunately, I do not possess enough knowledge to make head or tail of it except seeing the fact that the number of items in it increases, I would greatly appreciate it if someone could have a look when there's some free time on their hands

What is the expected behavior?

No response

What do you see instead?

No response

Additional information

I managed to isolate all parts of the code and the following is observed :->

If i comment out the mongodb call and say I hardcode and return the same payload, memory usage doesn't increase whatever the number of requests I send

Once it's added back memory usage continues to increase and doesn't drop

I used JMeter to spam requests to test this, but even without spamming only on normal production request rates the behaviour is the same

Akalanka47000 commented 3 weeks ago
image

This is the memory usage of an example container just in case it may be helpful. The 3 quick drops correspond to container restarts due to hitting the maximum memory limit that I've defined for now. This is within a span of 4 days

Akalanka47000 commented 3 weeks ago
image

This is another container, which was not restricted in memory

Akalanka47000 commented 3 weeks ago
image

This is another container, which was not restricted in memory

Apologies, ignore this scenario, tracked it down to a nasty bug in our code a few mins ago

Akalanka47000 commented 1 week ago

Hello, was there by any chance some time to look into this issue ? Here's a sample script which can reproduce it if that might be of any help ->

import mongoose from 'mongoose'

await mongoose.connect("<MONGO_URI>")

Bun.serve({
  port: 2005,
  async fetch(req) {
    const events = await mongoose.connection.collection("users").find().toArray()
    return new Response(JSON.stringify(events));
  },
})

FYI:

Also here are 3 outputs of heapStats() function at the following instances

  1. As soon as the server is up
{
  heapSize: 31349184,
  heapCapacity: 33725056,
  extraMemorySize: 23100096,
  objectCount: 96900,
  protectedObjectCount: 451,
  globalObjectCount: 1,
  protectedGlobalObjectCount: 1,
  objectTypeCounts: {
    GeneratorFunction: 8,
    ModuleProgramExecutable: 2,
    JSAsyncGeneratorFunction: 7,
    "Array Iterator": 1,
    "Set Iterator": 1,
    ProcessBindingConstants: 1,
    Int16ArrayPrototype: 1,
    require: 1,
    AsyncGeneratorFunction: 2,
    Function: 8918,
    Uint8ClampedArrayPrototype: 1,
    Request: 2,
    RegExp: 245,
    ShadowRealm: 1,
    JSSourceCode: 7,
    JSPropertyNameEnumerator: 10,
    UnlinkedFunctionExecutable: 7272,
    ModuleProgramCodeBlock: 2,
    FunctionExecutable: 7283,
    GlobalObject: 1,
    Math: 1,
    AsyncFromSyncIterator: 1,
    Intl: 1,
    Crypto: 2,
    WebAssembly: 1,
    AsyncGenerator: 10,
    StructureChain: 512,
    TextEncoder: 3,
    Dirent: 1,
    File: 1,
    StructureRareData: 1152,
    Blob: 1,
    Buffer: 1,
    Uint16Array: 6,
    Float64Array: 4,
    Response: 1,
    Uint16ArrayPrototype: 1,
    "WebAssembly.Module": 2,
    SymbolTable: 1746,
    Bun: 1,
    StringDecoder: 2,
    JSON: 1,
    Callee: 2,
    ProgramCodeBlock: 387,
    Uint32ArrayPrototype: 1,
    DebugHTTPServer: 2,
    CustomGetterSetter: 107,
    console: 1,
    "WebAssembly.Memory": 2,
    UnlinkedProgramCodeBlock: 420,
    ProgramExecutable: 387,
    Date: 12,
    DataView: 2,
    WebAssemblyFunction: 6,
    InternalModuleRegistry: 1,
    SubtleCrypto: 2,
    URL: 3,
    WeakRef: 1,
    CryptoHasher: 1,
    Int32ArrayPrototype: 1,
    ModuleLoader: 1,
    BigInt: 6,
    ModuleRecord: 7,
    PropertyTable: 837,
    String: 1,
    Map: 29,
    Boolean: 1,
    "Map Iterator": 1,
    TextDecoder: 2,
    InternalPromise: 16,
    "URLSearchParams Iterator": 1,
    Array: 11594,
    Process: 1,
    FinalizationRegistry: 1,
    Number: 1,
    ModuleNamespaceObject: 4,
    URLSearchParams: 2,
    Float64ArrayPrototype: 1,
    TLSSocket: 5,
    "Intl.ListFormat": 1,
    NativeExecutable: 796,
    DOMAttributeGetterSetter: 72,
    Prototype: 1,
    Float32ArrayPrototype: 1,
    UnlinkedFunctionCodeBlock: 1680,
    "WebAssembly.Instance": 2,
    FunctionRareData: 302,
    JSGlobalProxy: 1,
    JSGlobalLexicalEnvironment: 1,
    "Immutable Butterfly": 457,
    Set: 91,
    AsyncFunction: 366,
    Promise: 35,
    AsyncIterator: 1,
    NextTickQueue: 1,
    BigUint64ArrayPrototype: 1,
    BigInt64ArrayPrototype: 1,
    Arguments: 5,
    Structure: 11554,
    Uint8Array: 471,
    UnlinkedModuleProgramCodeBlock: 2,
    symbol: 280,
    Reflect: 1,
    ArrayBuffer: 12,
    EventEmitter: 1,
    ModulePrototype: 1,
    Performance: 2,
    JSModuleEnvironment: 8,
    InternalPromisePrototype: 1,
    Object: 2491,
    FunctionCodeBlock: 1731,
    "RegExp String Iterator": 1,
    Generator: 8,
    Int8ArrayPrototype: 1,
    ScopedArgumentsTable: 6,
    "Intl.DurationFormat": 1,
    Timeout: 7,
    Symbol: 1,
    WebAssemblyModuleRecord: 1,
    JSLexicalEnvironment: 1616,
    string: 16924,
    Iterator: 1,
    NodeJSFS: 2,
    Module: 443,
    Stats: 1,
    "String Iterator": 1,
    BufferList: 10,
    WeakMap: 3,
    CryptoKey: 2,
    InternalFieldTuple: 1,
    resolve: 1,
    EventTarget: 1,
    Uint8ArrayPrototype: 1,
    SparseArrayValueMap: 99,
    GetterSetter: 748,
  },
  protectedObjectTypeCounts: {
    UnlinkedModuleProgramCodeBlock: 2,
    AsyncFunction: 1,
    UnlinkedProgramCodeBlock: 420,
    GlobalObject: 1,
    DebugHTTPServer: 1,
    Function: 23,
    Timeout: 3,
  },
}
  1. After 4000 requests ( 4000 find calls)
{
  heapSize: 44730009,
  heapCapacity: 70790921,
  extraMemorySize: 31743577,
  objectCount: 198064,
  protectedObjectCount: 451,
  globalObjectCount: 1,
  protectedGlobalObjectCount: 1,
  objectTypeCounts: {
    GeneratorFunction: 8,
    ModuleProgramExecutable: 2,
    JSAsyncGeneratorFunction: 7,
    "Array Iterator": 1,
    "Set Iterator": 1,
    ProcessBindingConstants: 1,
    Int16ArrayPrototype: 1,
    require: 1,
    AsyncGeneratorFunction: 2,
    Function: 14688,
    Uint8ClampedArrayPrototype: 1,
    Request: 100,
    RegExp: 230,
    ShadowRealm: 1,
    JSSourceCode: 7,
    JSPropertyNameEnumerator: 9,
    UnlinkedFunctionExecutable: 7075,
    Math: 1,
    FunctionExecutable: 6768,
    GlobalObject: 1,
    AsyncFromSyncIterator: 1,
    Intl: 1,
    Crypto: 2,
    WebAssembly: 1,
    AsyncGenerator: 435,
    StructureChain: 518,
    TextEncoder: 3,
    Dirent: 1,
    File: 1,
    StructureRareData: 1566,
    Blob: 1,
    Buffer: 1,
    Uint16Array: 6,
    Float64Array: 4,
    Response: 99,
    Uint16ArrayPrototype: 1,
    "WebAssembly.Module": 2,
    JSON: 1,
    Bun: 1,
    StringDecoder: 2,
    SymbolTable: 994,
    Callee: 2,
    Uint32ArrayPrototype: 1,
    DebugHTTPServer: 2,
    CustomGetterSetter: 237,
    console: 1,
    "WebAssembly.Memory": 2,
    ProgramExecutable: 387,
    UnlinkedProgramCodeBlock: 420,
    Date: 5331,
    DataView: 245,
    WebAssemblyFunction: 6,
    InternalModuleRegistry: 1,
    SubtleCrypto: 2,
    URL: 3,
    WeakRef: 1,
    CryptoHasher: 1,
    Int32ArrayPrototype: 1,
    ModuleLoader: 1,
    BigInt: 71,
    ModuleRecord: 7,
    PropertyTable: 1337,
    String: 1,
    Map: 30,
    Boolean: 1,
    TextDecoder: 2,
    "Map Iterator": 1,
    InternalPromise: 16,
    "URLSearchParams Iterator": 1,
    Array: 21789,
    Process: 1,
    FinalizationRegistry: 1,
    Number: 1,
    ModuleNamespaceObject: 4,
    URLSearchParams: 2,
    Float64ArrayPrototype: 1,
    TLSSocket: 70,
    "Intl.ListFormat": 1,
    NativeExecutable: 800,
    DOMAttributeGetterSetter: 72,
    Prototype: 1,
    Float32ArrayPrototype: 1,
    "WebAssembly.Instance": 2,
    UnlinkedFunctionCodeBlock: 692,
    FunctionRareData: 485,
    JSGlobalProxy: 1,
    JSGlobalLexicalEnvironment: 1,
    "Immutable Butterfly": 441,
    Set: 91,
    AsyncFunction: 366,
    Promise: 2252,
    AsyncIterator: 1,
    NextTickQueue: 1,
    BigUint64ArrayPrototype: 1,
    BigInt64ArrayPrototype: 1,
    Arguments: 8,
    Uint8Array: 6564,
    Structure: 14119,
    UnlinkedModuleProgramCodeBlock: 2,
    symbol: 280,
    Reflect: 1,
    ArrayBuffer: 374,
    EventEmitter: 1,
    ModulePrototype: 1,
    Performance: 2,
    JSModuleEnvironment: 8,
    InternalPromisePrototype: 1,
    Object: 30291,
    FunctionCodeBlock: 1160,
    "RegExp String Iterator": 1,
    Generator: 1059,
    Int8ArrayPrototype: 1,
    ScopedArgumentsTable: 1,
    "Intl.DurationFormat": 1,
    Timeout: 10,
    Symbol: 1,
    WebAssemblyModuleRecord: 1,
    JSLexicalEnvironment: 5726,
    string: 29284,
    Iterator: 1,
    NodeJSFS: 2,
    Module: 443,
    Stats: 1,
    "String Iterator": 1,
    BufferList: 140,
    WeakMap: 3,
    CryptoKey: 2,
    InternalFieldTuple: 1,
    resolve: 1,
    Uint8ArrayPrototype: 1,
    EventTarget: 1,
    SparseArrayValueMap: 164,
    GetterSetter: 748,
  },
  protectedObjectTypeCounts: {
    UnlinkedModuleProgramCodeBlock: 2,
    AsyncFunction: 1,
    UnlinkedProgramCodeBlock: 420,
    GlobalObject: 1,
    DebugHTTPServer: 1,
    Function: 23,
    Timeout: 3,
  },
}
  1. 10 minutes after above 4000 requests ( Server is idle the whole time)
{
  heapSize: 49598041,
  heapCapacity: 71965625,
  extraMemorySize: 32901897,
  objectCount: 276971,
  protectedObjectCount: 451,
  globalObjectCount: 1,
  protectedGlobalObjectCount: 1,
  objectTypeCounts: {
    GeneratorFunction: 8,
    ModuleProgramExecutable: 2,
    JSAsyncGeneratorFunction: 7,
    "Array Iterator": 1,
    "Set Iterator": 1,
    ProcessBindingConstants: 1,
    Int16ArrayPrototype: 1,
    require: 1,
    AsyncGeneratorFunction: 2,
    Function: 18256,
    Uint8ClampedArrayPrototype: 1,
    Request: 100,
    RegExp: 230,
    ShadowRealm: 1,
    JSSourceCode: 7,
    JSPropertyNameEnumerator: 9,
    UnlinkedFunctionExecutable: 7075,
    Math: 1,
    FunctionExecutable: 6768,
    GlobalObject: 1,
    AsyncFromSyncIterator: 1,
    Intl: 1,
    Crypto: 2,
    WebAssembly: 1,
    AsyncGenerator: 1020,
    StructureChain: 518,
    TextEncoder: 3,
    Dirent: 1,
    File: 1,
    StructureRareData: 1577,
    Blob: 1,
    Buffer: 1,
    Uint16Array: 6,
    Float64Array: 4,
    Response: 99,
    Uint16ArrayPrototype: 1,
    "WebAssembly.Module": 2,
    JSON: 1,
    Bun: 1,
    StringDecoder: 2,
    SymbolTable: 994,
    Callee: 2,
    Uint32ArrayPrototype: 1,
    DebugHTTPServer: 2,
    CustomGetterSetter: 237,
    console: 1,
    "WebAssembly.Memory": 2,
    ProgramExecutable: 387,
    UnlinkedProgramCodeBlock: 420,
    Date: 6519,
    DataView: 1084,
    WebAssemblyFunction: 6,
    InternalModuleRegistry: 1,
    SubtleCrypto: 2,
    URL: 3,
    WeakRef: 1,
    CryptoHasher: 1,
    Int32ArrayPrototype: 1,
    ModuleLoader: 1,
    BigInt: 71,
    ModuleRecord: 7,
    PropertyTable: 1357,
    String: 1,
    ArrayIterator: 39,
    Map: 182,
    Boolean: 1,
    TextDecoder: 2,
    "Map Iterator": 1,
    InternalPromise: 16,
    "URLSearchParams Iterator": 1,
    Array: 35118,
    Process: 1,
    FinalizationRegistry: 1,
    Number: 1,
    ModuleNamespaceObject: 4,
    URLSearchParams: 2,
    Float64ArrayPrototype: 1,
    TLSSocket: 70,
    "Intl.ListFormat": 1,
    NativeExecutable: 800,
    DOMAttributeGetterSetter: 72,
    Prototype: 1,
    Float32ArrayPrototype: 1,
    "WebAssembly.Instance": 2,
    UnlinkedFunctionCodeBlock: 692,
    FunctionRareData: 680,
    JSGlobalProxy: 1,
    JSGlobalLexicalEnvironment: 1,
    "Immutable Butterfly": 630,
    Set: 103,
    AsyncFunction: 366,
    Promise: 4265,
    AsyncIterator: 1,
    NextTickQueue: 1,
    BigUint64ArrayPrototype: 1,
    BigInt64ArrayPrototype: 1,
    Arguments: 28,
    Uint8Array: 9226,
    Structure: 14300,
    UnlinkedModuleProgramCodeBlock: 2,
    symbol: 280,
    Reflect: 1,
    ArrayBuffer: 817,
    EventEmitter: 1,
    ModulePrototype: 1,
    Performance: 2,
    JSModuleEnvironment: 8,
    InternalPromisePrototype: 1,
    Object: 52262,
    FunctionCodeBlock: 1202,
    "RegExp String Iterator": 1,
    Generator: 1293,
    Int8ArrayPrototype: 1,
    ScopedArgumentsTable: 1,
    "Intl.DurationFormat": 1,
    Timeout: 205,
    Symbol: 1,
    WebAssemblyModuleRecord: 1,
    JSLexicalEnvironment: 8752,
    string: 35128,
    Iterator: 1,
    NodeJSFS: 2,
    Module: 443,
    Stats: 1,
    "String Iterator": 1,
    BufferList: 140,
    WeakMap: 3,
    CryptoKey: 2,
    InternalFieldTuple: 1,
    resolve: 1,
    Uint8ArrayPrototype: 1,
    EventTarget: 1,
    SparseArrayValueMap: 164,
    GetterSetter: 748,
  },
  protectedObjectTypeCounts: {
    UnlinkedModuleProgramCodeBlock: 2,
    AsyncFunction: 1,
    UnlinkedProgramCodeBlock: 420,
    GlobalObject: 1,
    DebugHTTPServer: 1,
    Function: 23,
    Timeout: 3,
  },
}
mrtime39 commented 1 week ago

Yes, when I use mongoose, even if there is not much data, the memory consumption is also very high.

Akalanka47000 commented 1 day ago

I happened to notice this as well, might help find the issue, the memory usage is stable if I connect to a local database, but keeps increasing if i connect to one on Atlas and also on another service i tried for testing named ScaleGrid

When i tested with the local database I configured ssl in it to check whether enabling ssl is causing the issue, though it seems that it is stable with both ssl enabled, though I'm not sure that it's behavior's the same as when connecting to atlas

Just to be fully sure, I tried building bun locally and trying to replace the tls module which seems to be implemented in Bun with Nodejs, though I wasn't able to get it done since I'm not really familiar with what's going on in the code and I don't have much time to keep at it anymore, please help out here if possible @Jarred-Sumner @cirospaciari