Question - dets:sync

[dergraf]

Currently the metadata dets is synced to disc for every put. Is this required or could it be disabled to speed up insertion of many objects?

[jrwest]

for a bit of background (note: I no longer work on this codebase on a regular basis): the use of dets:sync stems from this library's initial use within Riak. This code replaced an older system that guaranteed that updates were synced to disk after each write. This was important for Riak's use case but generally may not be wise.

A few options I see:

• add a sync flag and periodically sync regardless. This has some potential for dataloss but lets the user sync important data when needed, at the cost of performance.
• add a batch put, which would address this case directly but sync'ing only after all writes have been affected. Note: In conjunction with this it would probably be useful to be able to send a batch of updates over the broadcast layer instead of each individual update in the batch being a separate message to disseminate. Granted this method can be built on the former.
• get rid of dets (always was "on the roadmap") and use something safer (hand waves)

[tsantero]

I agree with @jrwest's comments. While delaying sync/using a batch put would decrease the cost of each write, the potential for dataloss makes plumtree that much less attractive.

For the foreseeable future I don't see replacing dets with anything else, as my immediate goals include further decoupling the peer service from the broadcast protocol and making CRDTs a first class citizen for objects.

I'd be happy to chat further offline, if you're interested. Feel free to drop me an email if you like: tom@helium.com

[dergraf]

Thank you both for your answers. Tom you've got mail.