search

cockroachdb / cockroach

CockroachDB - the open source, cloud-native distributed SQL database.
https://www.cockroachlabs.com
Other
29.51k stars 3.7k forks source link

sql: Inconsistent zone config behavior when dropping database or table #24179

Closed a-robinson closed 5 years ago

a-robinson commented 6 years ago

As "discovered" in #24154, we handle zone configs differently when dropping partitions/tables/databases.

  1. When you drop a table, we keep the corresponding zone config around until the table's descriptor is deleted, which is supposed to happen after the configured GC TTL period.
  2. I haven't tested dropping a partition, so I won't speak on it.
  3. When you drop a database, we delete the corresponding zone config immediately.

The behavior when dropping a database is concerning. If (a) there's data in a table in the database (b) the table doesn't have a table-level zone config (c) the database does have a database-level zone config, then dropping the database (e.g. DROP DATABASE ... CASCADE) has the effect of changing the zone config that applies to the data in the table. Before dropping the database, the databases's zone config applies to the table. After dropping the database, the global .default zone config is all that applies to the table.

This means that when we drop a database, it can actually cause a lot of rebalancing if the database had different constraints on it than the .default zone. Even if the database was constrained to certain regions for sovereignty reasons, the yet-to-be-GC'ed data will likely get moved elsewhere.

For example, I created a 6 node local cluster with 3 nodes in locality a=1 and 3 in a=2. I created a table, split it 20 times, and constrained all its replicas to a=1 around 16:07 in the below screenshot. At 16:09 I ran DROP DATABASE ... CASCADE on the database, and suddenly all the data got spread around across both localities because the constraint I had specified no longer existed.

screen shot 2018-03-23 at 12 13 46 pm

There may also be a similar problem with us deleting a table's zone config before all the data has been GC'ed. I'm not an expert in that area, though, so I'll let @benesch speak to that.

vivekmenezes commented 5 years ago

to be fixed after #19004 is fixed. Once a job exists for the DROP DATABASE schema change, the job can be used to drop the zone config later.