[docdb] PITR: Add persistent tracking for DDL statements

yugabyte / yugabyte-db

YugabyteDB - the cloud native distributed SQL database for mission-critical applications.

https://www.yugabyte.com

Other

8.92k stars 1.06k forks source link

[docdb] PITR: Add persistent tracking for DDL statements #8773

Open bmatican opened 3 years ago

bmatican commented 3 years ago

Jira Link: DB-1829 To start with, this can be a new type of sys_catalog entry, written to from every DDL entry point. We can use a simple time based query interface, to be able to see the last N entries / last M hours of entries, via yb-admin.

This will help for PITR, as we can then be able to rollback to right before executing a particular DDL statement:

For YCQL, the timestamp of the actual DDL record, written from CatalogManager, might be good enough.
For YSQL, we might want to integrate better with the PG side of tracking, to record the start and end of a DDL transaction.

bmatican commented 3 years ago

After the initial proof of concept, we should augment this with

enhance to track more types of DDLs, eg: create table is not tracked atm
client side filtering, eg: which record types, or which namespaces / tables are relevant, what time window
pagination, eg: LIMIT will be super important, to not overload the server on a request OR the client based on the RPC size, but at that point, we might as well have limit and offset
some debug interface, aside from just a yb-admin command, eg: either a master UI page, or RPCs that YW can call and build a UI on top of

cc @sanketkedia

sanketkedia commented 3 years ago

I am taking point on this. cc @spolitov to avoid duplication.

sanketkedia commented 3 years ago