Use cases

[danielsz]

Hi,

I've successfully implemented a durable queue according to the stated use case, as an in-process device that does both the puts and takes. However, initially I had a misgiving, and I had two processes, one doing the puts, the other the takes. This did not work. Despite the fact that the queue object was pointing to the same underlying file, the puts of one process did not result in tasks visible to takes in the second process. Now I am just curious, what are the technical reasons for this. Intuitively, I though that no further synchronization was needed beyond fsync to make this work.

Thank you in advance.

[ztellman]

I saw a link to this, and so I'll just answer this for posterity. The "state" of the task queue largely resides in-memory, with things persisted to disk just to make sure that the state can be recomputed by a linear scan of the entire task queue if the process every closes down.

To have separate processes which communicate via the task queue would require, at the very least, a way for the consumer process to be signaled whenever there's a new task available. There would also need to be some thought given as to who's responsible for GCing the on-disk slab files, and probably some other stuff I'm not remembering. It's all very doable, but wasn't necessary in the use case I built it for, and is unlikely to be added at this point.