search

go-graphite / carbon-clickhouse

Graphite metrics receiver with ClickHouse as storage
MIT License
186 stars 48 forks source link

Large discrepancy between cache size and incoming metrics rate #153

Closed spoofedpacket closed 1 week ago

spoofedpacket commented 1 week ago

I'm seeing some behaviour in the carbon-clickhouse cache which I can't explain, the cache size is almost 10x the size of the incoming metrics rate.

Previously, I have observed the inverse of this behaviour where the incoming metrics rate is higher than the cache, in that case it was caused by data points being reported multiple times in a one minute window.

We're using the default cache-ttl of 12h, running carbon-clickhouse 0.11.6-1

We have many clickhouse-based graphite instances and this is the only one which exhibits this behaviour, so there must be something peculiar about the data coming into this cluster which causes it. Does anyone have any pointers on how I'd go about investigating this ?

spoofedpacket commented 1 week ago

I realised we were receiving some metrics at 5 min or even close to 10 min intervals, so the cache size is actually legitimate and this explains the discrepancy between it and the incoming metrics rate. Closing this one off.