search

timescale / promscale_extension

[DEPRECATED] Tables, types and functions supporting Promscale
Other
37 stars 17 forks source link

Compression Job Improvements #576

Closed jgpruitt closed 1 year ago

jgpruitt commented 2 years ago

Description

Adds a view named metric_chunks_that_need_to_be_compressed which only returns results for metrics that have chunks which need to be compressed. It returns one row per metric with a jsonb array of objects listing the chunks to be compressed.

The execute_compression_policy previously looped over EVERY metric, grabbed an advisory lock on it, and then checked for chunks to be compressed for that metric. This new version uses the view above. In this way, it only loops over and locks metrics which have chunks to compress. Further, it does already has the list of chunks to compress. This reduces locking, transactions, and queries.

Merge requirements

Please take into account the following non-code changes that you may need to make with your PR:

jgpruitt commented 1 year ago

On a test system with the below stats, the query in _prom_catalog.metric_chunks_that_need_to_be_compressed generates the following explain plan

2,000 metrics 200,000 series 44,010 chunks

https://explain.dalibo.com/plan/a36861e6g71c741d

jgpruitt commented 1 year ago

On a test system with the below stats, the query in _prom_catalog.metric_chunks_that_need_to_be_compressed generates the following explain plan

2,500 metrics 250,000 series 1,496,370 chunks

https://explain.dalibo.com/plan/hd818fca8d52c9c8