webui: show protected timestamps in DB Console

[erikgrinaker]

Protected timestamps are used by various subsystems to "lock" MVCC history and prevent it from being garbage collected. For example, incremental backups need this to be able to back up the changes since the previous backup, and changefeeds need this in order to stream the changes that have happened since a changefeed paused or failed.

However, we've often seen such records get left behind indefinitely, such that we didn't run garbage collection for several months, increasing disk usage and reducing scan performance.

We should improve our observability of these protected timestamps. A few ideas:

• Expose relevant time series metrics
  • spanconfig.kvsubscriber.oldest_protected_record_nanos: the age of the oldest protected timestamp record, in nanoseconds
  • spanconfig.kvsubscriber.protected_record_count: the number of protected timestamp records
• Show a listing of protected timestamp records, obtained from system.protected_ts_records.
• Show which protected timestamps jobs are holding onto (join job ID with decoded system.protected_ts_records.meta).
  • Full list of PTS records and their keyspans in the job detail.
  • Number/age/spans of protected timestamps in job listing.

I'll leave it to @irfansharif and cluster observability to flesh out the details here. CDC and DR also have additional information/metrics/data about these protected timestamps, coordinate with them as well.

Jira issue: CRDB-28224

[irfansharif]

BTW, https://github.com/cockroachdb/cockroach/pull/98540/ added the oldest PTS record age to the console, right next the MVCC GC graphs.

[kevin-v-ngo]

Related issue: https://github.com/cockroachdb/cockroach/issues/96552

[kevin-v-ngo]

98540

Nice!!

[erikgrinaker]

Ah, my bad, should've searched better.