Open leonzz opened 3 years ago
I can see some value in providing a "strategy" for how the queue should behave in the event of inevitable data loss (discard old or new data), but the point of this library is to durably persist data so that it can be subsequently iterated over. Running out of disk space isn't really in the scope of the functionality provided.
Thanks @mreiferson Yes I understand it's out of the scope of the original design but with the added strategy of discarding old data, it'll enable many more use cases-)
If we have the bandwidth to implement this feature, would you consider accepting the PR and maintaining it later on? It will be guarded by a flag and won't break existing use cases.
Rather than building all this kind of functionality in to the core, I think that coming up with an API for diskqueue such that the application integrating with it could observe conditions like this and act on them however they wanted seems like a better approach.
For example, right now the API we expose only allows reading, writing, purging, and very minimal introspection (Depth()
). What we would need is to extend the API to expose more stats (like size on disk) and perhaps a way to purge specific segments.
Depth() is also costly to run; getting a less expensive way to observe queue depth would be great.
Right now new data
Put()
would fail if disk is out of space. It would be great to have an option to