Open saurako opened 7 years ago
@saurako Can you please enable debug and see if dispose message is generated below? https://github.com/dimichgh/mmap-kit/blob/d1bc006489ebed72cbfb756d7a67c4e85d85653c/lib/page-factory.js#L20-L27
You can reduce ttl to very small value and see if the file handlers are closed as you add more pages.
Thanks for responding so quickly, @dimichgh. I enabled debug and set the ttl to 30s, and I do see the dispose message being generated:
disposing page 0 from cache, file: meta_data/page-0.dat
disposing page 12 from cache, file: index/page-12.dat
disposing page 0 from cache, file: meta_data/page-0.dat
disposing page 12 from cache, file: index/page-12.dat
disposing page 0 from cache, file: meta_data/page-0.dat
disposing page 12 from cache, file: index/page-12.dat
disposing page 0 from cache, file: meta_data/page-0.dat
disposing page 12 from cache, file: index/page-12.dat
@saurako Then, maybe it works correctly? Are the files that you see still in active state as default ttl is 20 minutes?
@dimichgh - should they show up as deleted via lsof
if they were active? Not sure what's going on there.
The pages get deleted if they were not used or accessed in LRU cache for 20 minutes. Upon deletion, the file and its handler should be closed. The deletion is initiated when a new entry is put into the cache.
@dimichgh so essentially, the expectation is that disk usage should stay constant over time, if the rate of enqueue()
is the same as the rate of dequeue()
, correct?
@saurako right.
Thanks for confirming that, @dimichgh. I'd restarted the processes when I reported this issue. And started on a clean slate. Now I'm again seeing a constant increase in disk usage over the last couple days along with these unclosed file handles. There are only a few of them now since it's just been a couple days. Do you have any suggestions on how do I debug this? One thing I'd thought of is to reduce the ttl, and even if that works, it's only a workaround (but good for now).
Do you use default TTL? Do you have debug enabled to see what gets closed?
Yes, I do.
On Thu, Sep 14, 2017 at 6:31 PM Dmytro Semenov notifications@github.com wrote:
Do you use default TTL?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dimichgh/mmap-kit/issues/7#issuecomment-329652576, or mute the thread https://github.com/notifications/unsubscribe-auth/ACCsNOSq39cTLDV6jzjw-5XVrOAKbT7cks5sidOLgaJpZM4PVRV- .
@dimichgh is it necessary to call flush()
after enqueuing?
Hi, I've been using this module for a project I'm working on and it is a great fit for my needs. Really appreciate the work here. Over the course of a month of running a process that uses the BigQueue though, I've found that it leaves behind a trail of unclosed file handles. Here's an example:
Can you guys please help in fixing this? I'm using the BigQueue data structure, and simply
enqueue
anddequeue
messages from the queue. I do not explicitly close anything, but should I be doing that? If yes, then I'm not sure what should signal that I need to do that?