search

nsqio / go-diskqueue

A Go package providing a filesystem-backed FIFO queue
MIT License
471 stars 101 forks source link

Files marked .bad after "Too many open files" read error #1

Open rdingwall opened 7 years ago

rdingwall commented 7 years ago

The current diskqueue implementation does not discriminate between different types of errors returned by os.OpenFile(). We experienced a problem in production where an nsqd host ran out of file descriptors and os.OpenFile() returned a "Too many open files" error, so a number of NSQ diskqueue files were incorrectly marked as .bad, requiring manual repair (by renaming them and updating the cursor position in the metadata files). It would be good if NSQ was able to discriminate between transient "try again" read errors vs other more serious errors.

I don't have a PR to fix unfortunately, I just thought I would flag up.

mreiferson commented 7 years ago

Hi, thanks for the feedback.

I'm not sure what you would expect the behavior to be in that case. The only possible behavior I can think of that would avoid needing to manually repair files would be if nsqd refused to proceed until it was restarted — is that desirable vs. current behavior?

rdingwall commented 7 years ago

If nsqd encountered EMFILE or ENFILE on a diskqueue file, I would expect the behaviour to be something along the lines of:

I think?

mreiferson commented 7 years ago

Unfortunately, given nsqd's implementation (memory and disk) this is a little more complicated:

Client publishes a message: return an error to the client

Not all messages reach disk and so it would only make sense to error when a message "overflowed" to disk — which is basically how nsqd currently behaves.

New client subscribing: return an error to the client

Existing client subscription: either graceful recovery (subscription goes quiet until the file is readable again) or terminate the subscription and let the client reconnect

As per above, since there are commonly cases where most messages don't reach disk, it doesn't make sense to penalize all clients of the topic.

If all messages went to disk, e.g. if nsqd moved to a "log" style architecture, it would be far easier to make decisions about what appropriate behavior is for situations like this.