search

jjneely / buckytools

Go implementation of useful tools for dealing with Graphite's Whisper DBs and Carbon hashing
Other
87 stars 21 forks source link

Backfill/Rebalance process creates spikes on counters #40

Open rsommer opened 1 year ago

rsommer commented 1 year ago

We recently did a cluster migration within our graphite setup and used bucky rebalance to adjust data distribution afterwards. After the migration was done, we discoverd spikes in our counter-based graphs, all appearing at the hour-boundary. These spikes appeared only after the migration and only for backfilled counter data - all newly written datapoints are spike-less as they have been before. Example graph from our dashboards:

Screenshot from 2023-07-11 09-11-22

All the spikes have not been there before rebalancing took place and are gone for all datapoints that have been written since.

This was done using the last github release 0.3.2.

deniszh commented 1 year ago

Hi @rsommer

Do you using lock in your carbon / go-carbon? You can also try our fork of buckytools, we're using it in prod w/o any issues. But please use master version, we're not doing releases often.

rsommer commented 1 year ago

Hi @deniszh ,

indeed, flock was at its default value (false) - under normal conditions there is only go-carbon writing to the whisper files. Strange behaviour, though - because only Counts seem to be affected. I'll retry some cluster layout changes using your fork and flock set to true if I find some time.