Open vekterli opened 2 years ago
Sounds great to me.
I suggest changing the redundancy semantics to be like cloud when no distribution
is set.
@vekterli Where are we on this one ? Is everything for vespa 8 done ?
@baldersheim nothing has happened here yet. But the spirit is strong and willing.
When using multiple content cluster groups in self-hosted Vespa there is a requirement to specify a
distribution
attribute that states how replicas are to be distributed across groups (see https://docs.vespa.ai/en/performance/sizing-examples.html). This is a legacy feature primarily designed for the case where you might have a complex topology of nested groups, which was more of a use-case for the long-since removed VDS persistence provider, which did not support indexed search of any kind.Rationale
Syntax
The config syntax itself is far from obvious and may elicit a number of heartfelt questions from a user ("what numbers should I use?", "wildcards go where? what do they mean?", "why do I need a wildcard at the end?", "can dogs look up?" etc). There is nothing else in
services.xml
that uses this particular brand of mini-DSL.Semantics
The semantics are expertly (though of course not intentionally) engineered for maximum confusion and surprise (tm). In particular, the spec
2|*
with 2 groups and global redundancy of 3 does not, in fact, put 2 replicas in the first configured group and 1 in the second configured group. It puts 2 replicas in a group, more specifically the group that scored the highest during ideal state computation for the data bucket in question. It then assigns the remaining replica to the next lower scoring group (and so on, for each configured group).It is likely that an unsuspecting user trying this out would assume that their 2x sized group 0 would receive 2x the amount of data as their lesser sized group 1, only to be mildly bemused when they end up receiving (on average) the same amount of data, possibly blowing up some resource limits along the way.
This is so non-obvious that it effectively renders the feature moot by itself.
Usefulness
Indexed search and query load-balancing do not work well in highly heterogeneous setups, as they assume groups are fairly uniform in their performance, i.e. explicit partition spec is an enabler of anti-patterns.
Proposal
Vespa Cloud entirely hides the nitty-gritty details of the partition spec by enforcing that groups are homogenous and that they have equal intra-group redundancy. This is a good thing; one of the core responsibilities of a distributed system is to shield from the user the fact that the world is cruel and that computers secretly hate us. We should do something similar for the common case in self-hosted as well.
Thus:
On Vespa 8:
distribution
is set, fail ifredundancy
is not evenly divisible byn
. This is backwards compatible because we currently do requiredistribution
to be set when multiple groups are present.distribution
is explicitly set, emit a deprecation warning but otherwise allow what's configureddistribution
is not setOn Vespa 9 and beyond:
distribution
entirely and enforce uniform group redundancy.Up for bonus debate:
redundancy
in self-hosted to be equal to that of Vespa Cloud (i.e. per-group redundancy instead of global)? This would probably also be a Good Thing, but needs to be done with care.