How to store raw and aggregated data with different retentionPeriods

[yudelin]

          Stream-aggregation doesn't seem to meet my needs. My needs are:

1. There is no need to set aggregation rules, for example, there are 20,000 indicators in Prometheus, and I need to set all 20,000 indicators to avg, max, and min
2. I need to save the original data for 1 month, and the data for more than 1 month is aggregated, and this aggregation does not need to be configured. My current scenario is:
3. Use the VictoriaMetrics cluster to directly connect to the remote_write of prometheus to save one month's raw data
4. The second remote_rewrite of Prometheus is to aggregate min, max, and avg to a self-developed VMagent-like program every hour (this custom aggregation cycle). and set the label of the corresponding aggregate function to be added to the time series database. Write data to another VictoriaMetrics cluster, and the secondary cluster retains the data for 2 years
5. The promsql query distinguishes between the original data and the trend data, and corresponds to the aggregate function in the second step I would like to ask, is the above solution feasible? If feasible, how can retentionPeriod distinguish between the retention period of the two parts of the data, the original data is stored for 1 month, and the aggregated data is stored for 2 years. Can this only be achieved by deploying two VictoriaMetrics clusters?

Originally posted by @yudelin in https://github.com/VictoriaMetrics/VictoriaMetrics/issues/7010#issuecomment-2367407406

[Haleygo]

Hello!

how can retentionPeriod distinguish between the retention period of the two parts of the data, the original data is stored for 1 month, and the aggregated data is stored for 2 years.

It's feasible with retention filters in the ENT version of VictoriaMetrics. With OSS, you do need two VictoriaMetrics clusters with different retentionPeriod. So the first cluster is for storing the raw data, and second one is for long-term aggregated data. And there are two ways to write aggregated data to the second VictoriaMetrics cluster:

1. vmagent stream aggregation. Add both VictoriaMetrics clusters as remoteWrite destinations, write data directly to the first VictoriaMetrics, and add -remoteWrite.streamAggr.config only for the second VictoriaMetrics.
2. vmalert recording rule. Create a vmalert which uses the first cluster as datasource, performs queries and writes the aggregated results to the second cluster.

The second remote_rewrite of Prometheus is to aggregate min, max, and avg to a self-developed VMagent-like program every hour (this custom aggregation cycle).

stream aggregation is more suitable if the raw data can be dropped to reduce the storage load, which I assume is not your case here. And since you only want the min, max, and avg calculations per hour. I'd recommend to go with the vmalert option, which is more lightweight in this case

[yudelin]

The vmalert recording rule here does not seem to meet the existing requirements, if you want to use the vmalert recording rule, then you need to configure a large number of rules, for example, the number of labels in the existing tsdb is 1 million, then you need to configure 1 million rules, what I need now is to directly aggregate the underlying data, for example: {"a":"b","c":"d"} has monitoring data as 1 2 3 three values, After aggregation, the data sent is {"a":"b","c":"d","func":"avg"},{"a":"b","c":"d","func":"min"},{"a":"b","c":"d","func":"max"},,the values correspond to 2,1,3 respectively,I have now written a program with golang,vmagent I have seen。 Not quite in line with the current use case. Another problem is that if three data are formed, the func need to be specified when the promsql expression is formed, so that the promsql needs to be modified. Is there a better way? If you need to write a large number of vmalert recording rules in your solution, is there a way that you don't need to write vmalert recording rules?