Add docs on how CockroachDB handles "split-brain" scenarios (network partitions)

[rmloveland]

Richard Loveland (rmloveland) commented:

Per this forum conversation, the behavior is as follows:

In a split brain scenario ongoing operations are suspended and wait/block until the partition resolves. There are no writes during the partition (they are blocked), so we don't need to re-sync. The block is only for the minority side, so the majority side can proceed. Writes don't get committed until a majority has acknowledged them. If the acknowledgement went through, the write is ok; if the acknowledgement is blocked, the write is considered aborted. Reads that encounter a write that's too recent (i.e. we're not sure 100% that it's not going to abort) need to wait, or retry, or push the write forward in time. Ongoing operations resume when the partition resolves.

We should probably update the network partition troubleshooting docs with some of this info. It could also perhaps link to/from the bounded staleness reads stuff

Jira Issue: DOC-1140

[taroface]

Hi @thtruo @mwang1026 , do we consider this more of an Observability or a Disaster Recovery issue?

cc @kathancox

[mwang1026]

I actually consider this more OX since OX owns network and behavior in light of network issues. cc @piyush-singh :)

Happy to take this offline too given that so much of the solution is visibility into partitions (and to avoid a hot potato problem)

for the record, this is "resilience" IMO and no "disaster" (my personal opinion)

[piyush-singh]

Agreed, I think this should live with the docs on the network latency matrix or similar where we describe how to identify a network partition. If we want to have a dedicated page on network partitions, we could split that out, but it should still be either Observability or Deployments and Ops (DB Server) in my opinion.