search

ydb-platform / ydb

YDB is an open source Distributed SQL Database that combines high availability and scalability with strong consistency and ACID transactions
https://ydb.tech
Apache License 2.0
4k stars 565 forks source link

Snapshots for schema blobs in base tablet #3341

Open vporyadke opened 7 months ago

vporyadke commented 7 months ago

Blobs with schema updates are never freed and are kept forever in a tablet's history. This issue is for fixing that.

vporyadke commented 7 months ago

https://github.com/ydb-platform/ydb/commit/d127a0339c7b3518cbe4d5f52cd309dbc57ba2da

How it works:

  1. Check for schema blobs in previous storage groups on tablet start.
  2. If any are present, do an alter commit dumping the entire schema with a new "rewrite" flag
  3. When making a snapshot after that, add outdated blobs to GcSnapLeft & do not add them to SchemeInfoBodies
  4. When recovering from a snapshot made prior to (3), read the rewrite flag. This way the state after such recovery is the same as the state after (2).
  5. When recovering from a snapshot made in (3), outdated schema blobs are not accessed (since they are not in SchemeInfoBodies) and are deleted (since they are in GcSnapLeft)

Compatability:

Rollback after (2) is safe: the commit is treated as a no-op by old verisons. The flag will be read by the new version, when we then upgrade to it again, but that doesn't really matter, as it will do a new one anyway.

Rollback before (4) is safe: the snapshot in (3) is not confirmed yet and is ignored, everything is the same as rolling back right after (2)

Rollback before (5) is safe: the behavior in (5) is unchanged between versions