Add NoSync option

[tylertreat]

Allow users to make a trade-off between performance and safety by exposing an option to disable fsync on the database. Also add a Sync call to allow explicit fsyncs.

[armon]

@tylertreat The risk is that Raft will "forget" transactions after a crash/power loss if syncing is disabled. For use cases that are okay with those semantics, we generally recommend putting BoltDB on a RAM disk.

To optimize for performance, we are planning on a new backend that is an actual write ahead log, instead of a B-Tree, which should significantly improve write throughput for Raft on spinning disk!

[tylertreat]

@armon Understood on the risk. My thought is if we are replicating to enough nodes, the replication itself is sufficient for HA of data since likelihood of more than a quorum of nodes failing is low (similar to what Kafka does for its replication).

The WAL backend sounds interesting. Is there a repo to follow or is that just planned work? Any chance the WAL would allow for avoiding dual writes (one to Raft log and one to FSM storage)? This is a problem I currently have, and I'm considering writing my own LogStore/StableStore implementation.

[armon]

@tylertreat If you are comfortable with that risk, we can merge this. The WAL backend is just planned at this point, but it would not solve the dual writes. The FSM storage typically is a more summarized format, so it's hard to reconcile the two (at least in the case of Nomad/Consul).

[tylertreat]

Thanks!

[komuw]

@armon hi, re: To optimize for performance, we are planning on a new backend that is an actual write ahead log, instead of a B-Tree

has this been implemented somewhere I can have a look at?

thanks.

[armon]

@komuw We haven't yet implemented it unfortunately! SSDs give us an easy out, so it's been a low priority issue.

[komuw]

I see. thanks. SSDs FTW