search

monogon-dev / NetMeta

NetMeta is a scalable network observability toolkit optimized for performance.
https://netmeta.demo.monogon.dev/
Apache License 2.0
135 stars 6 forks source link

Downsampling for long-term storage #46

Open leoluk opened 4 years ago

leoluk commented 4 years ago

We don't need 1-second high resolution data for long-term archival.

Use an AggreatingMergeTree, aggregate it to 5m or so and possibly keep min/max/median.

fionera commented 1 year ago

This would require flows_raw to be migrated to the new table type OR we create a second table and use a view. If we do the queries correct there shouldnt be any overhead