Configurable total size limit and delete old data when exceeded

[leonzz]

Right now new data Put() would fail if disk is out of space. It would be great to have an option to

1. limit the total disk space consumed by diskqueue;
2. delete old data when space limit is reached but we still have new data coming in.

[mreiferson]

I can see some value in providing a "strategy" for how the queue should behave in the event of inevitable data loss (discard old or new data), but the point of this library is to durably persist data so that it can be subsequently iterated over. Running out of disk space isn't really in the scope of the functionality provided.

[leonzz]

Thanks @mreiferson Yes I understand it's out of the scope of the original design but with the added strategy of discarding old data, it'll enable many more use cases-)

If we have the bandwidth to implement this feature, would you consider accepting the PR and maintaining it later on? It will be guarded by a flag and won't break existing use cases.

[mreiferson]

Rather than building all this kind of functionality in to the core, I think that coming up with an API for diskqueue such that the application integrating with it could observe conditions like this and act on them however they wanted seems like a better approach.

For example, right now the API we expose only allows reading, writing, purging, and very minimal introspection (Depth()). What we would need is to extend the API to expose more stats (like size on disk) and perhaps a way to purge specific segments.

[iseletsk]

Depth() is also costly to run; getting a less expensive way to observe queue depth would be great.

[leonzz]

Thanks for the follow-ups! We've implemented a disk size limit in this fork It added new matadata (e.g. number of msg per file) to keep accurate track of depth when a file is deleted before full read. Feel free to check it out and merge to this main repo if you like-)