search

joncrlsn / dque

dque is a fast, embedded, durable queue for Go
MIT License
767 stars 46 forks source link

Segment files are not closed explicitly #10

Closed neilisaac closed 4 years ago

neilisaac commented 4 years ago

If the queue grows faster than it is consumed, Enqueue will append new segments. newQueueSegment opens segment files with os.O_APPEND|os.O_CREATE|os.O_WRONLY but they are only closed (a) if they become the firstSegment and get fully consumed and removed via delete in Dequeue, or (b) closed by the os.File finalizer. This may result in many open file descriptors until the garbage collector runs.

joncrlsn commented 4 years ago

This seems like a problem, and I'm glad you spotted it. When it creates a new segment, it should check to see if the old segment is also the first segment. If not, it should be closed.

joncrlsn commented 4 years ago

So I wrote a fix for this and I pushed it to the v2 branch instead of a development branch (oops, but I think I tested it pretty well).

I suspect I didn't catch this because I had a high open file limit when I wrote this. But now that I've reinstalled the OS on my Mac I get "too many open file" problems without this code. Can you pull and review my changes to the Enqueue() method?

neilisaac commented 4 years ago

Change looks good to me.