Open leventov opened 3 years ago
On the point of the inefficient data layout, partitioning/segmenting the data points by the series would also allow to throw away the index by series_id
on the main table, which contributes almost a half to the total disk usage by this table.
Hi @leventov pg_prometheus has been sunset for the new system released here, which should be much more performant, has native PromQL support, native compression, and more. It has a very different internal data model that the older version (which was released some 3 years ago).
The design with two tables:
values
andlabels
is suggested here, in the "Pg_Prometheus" section.Schema
The part of our implementation of this design, corresponding to the "values" table in the documentation:
Our data has about 250 unique series, all with 1 data point per second, i. e. about 250 data points per second in total.
Problem no. 1 - Hash Join when selecting just 190 indexed series_id is suboptimal
Query planner decided to do a Hash Join with just 192 rows from the
series
table, which doesn't make sense.When I replace the nested
SELECT
query withseries_id IN (21,22,...212)
, i. e. literally all the selected values, Timescale does the same (I don't post the query and the result because the query string is very long).But I can make Timescale to do an Index Scan instead of a Hash Join by using a
series_id between 21 and 212
clause. This is 40% faster:Problem no. 2 (deeper) - data layout is inefficient
If you look carefully at the plan of the query above (taking 14 seconds), you realise that it's still utterly inefficient for the data problem at hand. Data points belonging to the same
series_id
are about 250 rows away from each other, all interleaved in the main table (because there are about 250 series, and they all have the same frequency of 1 second). So the Index Scan is much closer to random access than to sequential data access to the columns in theseries_values
table.Prometheus does the same query on the same data in just 1.5 seconds, i. e. an order of magnitude faster. I this is because Prometheus always groups the data points by the metric. Timescale can do the same, but only on the compressed chunks, via the
compress_segmentby
option. (Note that the sole chunk appearing in the queries above is uncompressed.)Another way to group the data points belonging to the same series in Timescale is partitioning via
add_dimension()
. However, this solution is actively discouraged in the docs:I didn't try this yet, but indeed I doubt that PostgreSQL/Timescale will always do projection pushdowns properly, and that query planning will not become super slow with thousands of chunks per table. In fact, we already had problems with query planning latency with just hundreds of chunks (by time only), so we had to increase the chunking interval.