write consistency levels -- quorum of two is two

[shikhar]

In TransportShardReplicationOperationAction's verification of the write consistency level where it checks before executing the write if sufficient shards are available,

                    int requiredNumber = 1;
                    if (consistencyLevel == WriteConsistencyLevel.QUORUM && shardIt.size() > 2) {
                        // only for more than 2 in the number of shardIt it makes sense, otherwise its 1 shard with 1 replica, quorum is 1 (which is what it is initialized to)
                        requiredNumber = (shardIt.size() / 2) + 1;
                    } else if (consistencyLevel == WriteConsistencyLevel.ALL) {
                        requiredNumber = shardIt.size();
                    }

the (n / 2) + 1) should apply even with just 2 shards, giving a requiredNumber = 2. This is the commonly accepted definition.

[clintongormley]

It would stop Elasticsearch from working out of the box, as you'd have to run at least two nodes before you could index anything.

[shikhar]

@clintongormley is the default number_of_replicas not 0? then that's the problem :)

[shikhar]

note that with 1 shard, in java (1/2)+1) results in the right thing == 1

[shikhar]

I see that the default number_of_replicas is 1. So to have an index on just one node and keep cluster health green, you anyway have to set this to 0.

[clintongormley]

Having a yellow cluster in dev is just fine. Having a good default for number_of_replicas (ie 1) is good. Not being able to develop on Elasticsearch without running two instances is bad.

What's the worst that could happen? The primary accepts a write where there isn't a replica available. It's not like a minority of replicas is accepting writes.

[shikhar]

Having a yellow cluster in dev is just fine. Having a good default for number_of_replicas (ie 1) is good. Not being able to develop on Elasticsearch without running two instances is bad.

Going to production, having two instances, number_of_replicas = 1 (the default), using the write consistency level of quorum (also the default), and seeing data loss because one node went down -- is way worse.

What's the worst that could happen? The primary accepts a write where there isn't a replica available. It's not like a minority of replicas is accepting writes.

1 is not a quorum of 2. The point of a write consistency level of quorum is (or should be) to have redundancy if one node was to go down. It's misleading.

[kimchy]

I disagree, this default is good out of the box, otherwise ES will not work on a single node case with default settings. We can potentially document it better, that this is what happens when there is 1 replica (so total of 2 copies of the data).

[shikhar]

The out of the box experience can be addressed by changing the default to number_of_replicas = 0. It's really not a big deal to have to change that setting for production. In case users want a green cluster health with the current default, they have to setup discovery for the additional node in any case.

WriteConsistencyLevel.QUORUM is really WriteConsistencyLevel.QUORUM_BUT_NOT_ALWAYS. IMO it's not really something that should be papered over with a warning in the documentation.

[kimchy]

I think that having the default as 1 for number of replicas is a better out of the box solution. Seems like we are at an impasse and rehash the same argument, this is the decision we went with at Elasticsearch, and changing it have too many downsides, with no real upsides.

[shikhar]

Impasse acknowledged :) I will open a separate issue for the real problem, which is that acknowledged writes may be lost since the current consistency-level check only happens before the write is issued (based on my current understanding anyway, will verify...)

[kimchy]

FYI, I updated the docs to reflect this case, we missed it on our end.

I will add, that the number 2 is just a tricky number when it comes to distributed systems. I would argue that either quorum in this case set to 2, or it being set to 1 can be debatable..., since 2 in this case also means all. The reason we went with the default mentioned is because many times people run Elasticsearch using 1 node, or 2 (as a search platform for their database), on top of the just getting started aspect, and they are ok with potentially needing to reindex the data with the downsides that come with 1 or 2 nodes.

Its similar in nature to deciding what should be the default number for minimum_master_nodes when running with 2 nodes. Any value you choose for it has implications, 1 means you might loose data and hit split brain, 2 means that once a single node is down, you can't do anything with the cluster (in our improve zen branch, we added the ability where they can opt to still being able to search on it, which will help the decision of which value to choose). For people running Elasticsearch with only 2 nodes, they need to decided and live with the implications that come with it.

The same applies to running Elasticsearch with 2 copies of the data (number of replicas set to 1). If people care more about their data, then they will configure it to run with 2 replicas (3 copies), and then the default quorum does the right thing. But many are ok with running with 2 copies of the data (for the scenario explained above), and not have to allocate more resources to sustain a cluster with 3 copies of the data. Its a matter of tradeoffs, and the number 2 just makes it tricky....

Maybe the name quorum is misleading when it comes to 2 copies of the data, but to be honest, I haven't seen users being confused by our default behavior, and they understand what it means. So it doesn't seem beneficial to change the current behavior, especially when it can be easily configured (config or per API)

[Wilfred]

FWIW, as a user, I was confused by this issue: https://github.com/mobz/elasticsearch-head/issues/134

[kimchy]

@Wilfred I don't think that its the same type confusion, this one is more of "why my cluster is yellow" if I have one node, which is because ES defaults to 1 additional replica. If it defaults to 2 additional replicas, then the cluster would not be writable with 1 node.

[shikhar]

A scenario where this led to actual trouble for a user: https://groups.google.com/d/msg/elasticsearch/M17mgdZnikk/N6q9iGWRxncJ

[shikhar]

v2.0 seems like a good opportunity to fix this :-)

The main argument I have heard here is the out-of-the-box experience, which can be addressed by making the default number_of_replicas 0.

[bleskes]

@shikhar the one node experience is one argument. The other is being fault tolerant when loosing a node (and it's replicas). If we remove the exception of quorum for the two data copies case, (i.e., primary and replica) people won't be able to index when they loose a node, until the replica is re-constructed on another node - which may take a long time depending on shard size. To avoid this, one would need to keep 3 copies of everything by default. This is all about making the right tradeoffs between things and we feel that asking people to have 3 copies of their data (potentially many terabytes if not more) by default goes too far. The alternative of having only one copy by default also doesn't seem appealing as it is a risky default. Of course, people can make another choice and change the setting - and we should document that clearly.

[dakrone]

It's been a year of discussion on this since the last comment, I don't think we are planning on changing the default number of replicas to 0, or changing the quorum value, so I'm going to close this, we can re-open if people still want to discuss it.

[bleskes]

Thx @dakrone for bubbling this up. I think this is covered by #19454, which removes the quorum terminology, so we can consider it properly closed.