search

dimichgh / mmap-kit

nodejs memory mapped file
Other
10 stars 3 forks source link

Unclosed file handles #7

Open saurako opened 7 years ago

saurako commented 7 years ago

Hi, I've been using this module for a project I'm working on and it is a great fit for my needs. Really appreciate the work here. Over the course of a month of running a process that uses the BigQueue though, I've found that it leaves behind a trail of unclosed file handles. Here's an example:

$ sudo lsof -n | grep -i "DEL"
node       4291           saurako  DEL       REG              202,1               394612 /home/saurako/data-processor/data/page-66.dat
node       4291           saurako  DEL       REG              202,1               394610 /home/saurako/data-processor/data/page-64.dat
node       4291           saurako  DEL       REG              202,1               394611 /home/saurako/data-processor/data/page-65.dat
node       4291           saurako  DEL       REG              202,1               394557 /home/saurako/data-processor/data/page-62.dat
node       4291           saurako  DEL       REG              202,1               394555 /home/saurako/data-processor/data/page-61.dat
node       4291           saurako  DEL       REG              202,1               394558 /home/saurako/data-processor/data/page-63.dat
node       4291           saurako  DEL       REG              202,1               394554 /home/saurako/data-processor/data/page-60.dat
node       4291           saurako  DEL       REG              202,1               394552 /home/saurako/data-processor/data/page-58.dat

Can you guys please help in fixing this? I'm using the BigQueue data structure, and simply enqueue and dequeue messages from the queue. I do not explicitly close anything, but should I be doing that? If yes, then I'm not sure what should signal that I need to do that?

dimichgh commented 7 years ago

@saurako Can you please enable debug and see if dispose message is generated below? https://github.com/dimichgh/mmap-kit/blob/d1bc006489ebed72cbfb756d7a67c4e85d85653c/lib/page-factory.js#L20-L27

You can reduce ttl to very small value and see if the file handlers are closed as you add more pages.

saurako commented 7 years ago

Thanks for responding so quickly, @dimichgh. I enabled debug and set the ttl to 30s, and I do see the dispose message being generated:

disposing page 0 from cache, file: meta_data/page-0.dat
disposing page 12 from cache, file: index/page-12.dat
disposing page 0 from cache, file: meta_data/page-0.dat
disposing page 12 from cache, file: index/page-12.dat
disposing page 0 from cache, file: meta_data/page-0.dat
disposing page 12 from cache, file: index/page-12.dat
disposing page 0 from cache, file: meta_data/page-0.dat
disposing page 12 from cache, file: index/page-12.dat
dimichgh commented 7 years ago

@saurako Then, maybe it works correctly? Are the files that you see still in active state as default ttl is 20 minutes?

saurako commented 7 years ago

@dimichgh - should they show up as deleted via lsof if they were active? Not sure what's going on there.

dimichgh commented 7 years ago

The pages get deleted if they were not used or accessed in LRU cache for 20 minutes. Upon deletion, the file and its handler should be closed. The deletion is initiated when a new entry is put into the cache.

saurako commented 7 years ago

@dimichgh so essentially, the expectation is that disk usage should stay constant over time, if the rate of enqueue() is the same as the rate of dequeue(), correct?

dimichgh commented 7 years ago

@saurako right.

saurako commented 7 years ago

Thanks for confirming that, @dimichgh. I'd restarted the processes when I reported this issue. And started on a clean slate. Now I'm again seeing a constant increase in disk usage over the last couple days along with these unclosed file handles. There are only a few of them now since it's just been a couple days. Do you have any suggestions on how do I debug this? One thing I'd thought of is to reduce the ttl, and even if that works, it's only a workaround (but good for now).

dimichgh commented 7 years ago

Do you use default TTL? Do you have debug enabled to see what gets closed?

saurako commented 7 years ago

Yes, I do.

On Thu, Sep 14, 2017 at 6:31 PM Dmytro Semenov notifications@github.com wrote:

Do you use default TTL?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dimichgh/mmap-kit/issues/7#issuecomment-329652576, or mute the thread https://github.com/notifications/unsubscribe-auth/ACCsNOSq39cTLDV6jzjw-5XVrOAKbT7cks5sidOLgaJpZM4PVRV- .

saurako commented 7 years ago

@dimichgh is it necessary to call flush() after enqueuing?