Feature: retention time marker procedure - Githubissues

datafuselabs / databend

𝗗𝗮𝘁𝗮, 𝗔𝗻𝗮𝗹𝘆𝘁𝗶𝗰𝘀 & 𝗔𝗜. Modern alternative to Snowflake. Cost-effective and simple for massive-scale analytics. https://databend.com

https://docs.databend.com

Other

7.65k stars 726 forks source link

Feature: retention time marker procedure #5518

Open dantengsky opened 2 years ago

dantengsky commented 2 years ago

Summary

Provides a way of marking historical snapshots invisible, so that the old snapshots( and maybe the data it referenced) can fade away gradually.

Basic desc of functionalities:

Marks the latest visible snapshot of the given table.

A system configuration, let's say table_retention_time: Duration,
A system procedure, which marks the latest visible snapshot of a table
- by insert/update a specified key of the KV service
- TimeTravel of table data will respect this mark

NOTE: The query nodes work on their local clocks, which is NOT perfectly synced

basic idea of impl:

provides a system procedure, let's say call system$retention_mark([database_name,] table_name)
- grab meta data of the table specified
- check if key LATEST_VISBLE_SNAPHOST of the give table exist LATEST_VISBLE_SNAPHOST/<tid> -> timestamp
- if it exist and value of it is less than (now() + table_retention_time) try to update it to (now() + table_retention_time)
- if it does not exist try to insert the kv pair
- And of course, the mutations should be executed in a kv transaction
- the most important invariant of this operation value of LATEST_VISBLE_SNAPHOST/<tid> should only be increased

Notes:

if database_name is not provided, use the context's current database name
A "hurry" marker, whose clock is crazily ahead of time, may mark the LATEST_VISBLE_SNAPHOST "incorrectly"
- Have to live with it, hoping it is not too crazy : )
  
  e.g. if the clock is two months ahead of time. The history of the table may be not accessible in the next 2 months.
  
  To intimidate this situation: The value of LATEST_VISBLE_SNAPHOST/<tid> could be changed to the timestamp of the snapshot, by navigating to the snapshot S at (now() + table_retention_time). Thus, snapshots generated after S, could be accessible, if clocks go back to normal.
- The "current snapshot" referenced by the KV meta, is always visible

dantengsky commented 10 months ago

being addressed, as part of the new vacuum impl. pls refer:

BohuTANG commented 3 months ago

Can this issue be closed?