server: default zone config is not propagated into virtual clusters

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.

https://www.cockroachlabs.com

Other

29.89k stars 3.77k forks source link

server: default zone config is not propagated into virtual clusters #110003

Open knz opened 1 year ago

knz commented 1 year ago

Found while working on #110001.

Describe the problem

Found while investigating a test failure that was looking at the default zone config (the DefaultZoneConfig field in the system config provider).

The DefaultZoneConfig for secondary tenant services is initialized to the global default. It does not inherit from the KV layer (it should).

This default is used in two places:

when running ALTER RANGE default CONFIGURE ZONE USING DEFAULT if the KV layer has num replicas = 1, then this statement unexpectedly will set the tenant default config to num replicas = 3.
when using the .DefaultZoneConfig() accessor in ApplicationLaterInterface in test servers.

Expected behavior

ALTER RANGE default CONFIGURE ZONE USING DEFAULT should always use the default tenant zone config as set via the system tenant.

Epic CRDB-26686

Jira issue: CRDB-31213

knz commented 1 year ago

The resolution here is probably:

a new virtual range name alongside "liveness", "meta" etc in the system tenant, that holds the default zone config for secondary tenants.
a watcher for changes to it, that pumps the zoneconfig protobuf into the KV side of the tenant connector
receiving changes to this in the tenant connector
applying them to the server config tenant-side.

yuzefovich commented 11 months ago

Based on the discussion here, the second place .DefaultZoneConfig() is separate from RANGE default, so I don't think the second point requires any action. The first point about propagating RANGE default zone config from system tenant into virtual clusters is still valid though.

yuzefovich commented 11 months ago

@knz I need a bit more guidance from you. As it turns out, there is also pre-defined RANGE TENANTS (introduced in #86343) which is used as the initial value of RANGE DEFAULT for newly-created virtual clusters. Thus, I'm a bit confused about the desired end state here. I'm my reading of the issue is correct, we would like for virtual clusters' RANGE DEFAULT to be effectively RANGE DEFAULT of the system tenant, and these two would be reconciled via a new watcher. However, how does RANGE TENANTS fit into this? Should it be deprecated?

knz commented 11 months ago

We sketched the direction in this doc.

stevendanna commented 2 months ago

DB Server to eventually consider the work already done here and decide on a direction.