kriszyp / lmdb-js

Simple, efficient, ultra-fast, scalable data store wrapper for LMDB
Other
505 stars 42 forks source link

Garbage collection is not triggered with `cache: true` #45

Closed vladar closed 3 years ago

vladar commented 3 years ago

Some context first - occasionally we need to iterate over big ranges of keys and fetch value by key on each step. Without cache: true we see the expected behavior of garbage collection - unused objects are released during iteration.

But when I enable caching - memory keeps growing and nothing seems to be garbage-collected during iteration. And even after forced gc - something is retained in memory.

Not sure if it is a bug or expected behavior but if it is expected - then it is not entirely clear when garbage collection actually kicks-in.

Here is a test script: https://gist.github.com/vladar/046bae076d8d42f802fb520e7ff8b226

Running without cache (node --expose-gc gc-test.js):

heap: 7.02 MB / 12.56 MB
heap: 6.82 MB / 12.56 MB
heap: 7.53 MB / 12.81 MB
heap: 4.21 MB / 12.81 MB
heap: 4.82 MB / 12.81 MB
heap: 5.44 MB / 12.81 MB
heap: 6.03 MB / 12.81 MB
heap: 6.62 MB / 12.81 MB
heap: 7.22 MB / 13.06 MB
heap: 7.84 MB / 13.06 MB

after loop:
heap: 4.42 MB / 13.06 MB
after forced gc:
heap: 3.04 MB / 12.56 MB

Running with cache (CACHE=1 node --expose-gc gc-test.js):

heap: 35.64 MB / 58.22 MB
heap: 39.45 MB / 58.22 MB
heap: 37.16 MB / 60.22 MB
heap: 42.03 MB / 60.22 MB
heap: 45.55 MB / 66.47 MB
heap: 46.01 MB / 66.63 MB
heap: 46.06 MB / 66.63 MB
heap: 46.09 MB / 66.63 MB
heap: 46.13 MB / 66.63 MB
heap: 46.48 MB / 66.95 MB

after loop:
heap: 46.53 MB / 66.95 MB
after forced gc:
heap: 43.19 MB / 78.57 MB
vladar commented 3 years ago

I didn't realize that the cache option could be an object with all the options from https://github.com/kriszyp/weak-lru-cache

By setting cache: { cacheSize: number } or cache: { expirer: false } I can control the behavior. Sorry for the noise.