Carbon is one of the components of Graphite, and is responsible for receiving metrics over the network and writing them down to disk using a storage backend.
We've had this on a moderately sized production system for some time now and haven't spotted any problems. I started with work on #888 by @ploxiln and the original work by @piotr1212, but this approach is slightly different, and happens to have a slightly more readable diff. Differences:
We don't drop excessive creates, instead we just queue them like with updates. This matters a lot on our setup, where large numbers of metrics often get created in bursts, and we'd rather not lose data from them.
It's (more) robust to database files getting deleted under its hood. This will cause a droppedCreate rather than an exception, I think.
There are a few behaviour changes:
We'll create new metrics at the maximum rate possible until there are none left to create. The create bucket should likely therefore be tuned not to be able to overwhelm the write bandwidth -- though if this happens, the cache is going to be unable to keep up anyway.
If the update bucket is empty, this can delay creates, as the thread can be blocked there for one update even if a create is possible first. This is unlikely to delay creates by very much.
Empty Whisper files will now be created, as the create step doesn't write data into the resulting file.
Creates are now dropped only when the cache tries to fill the new metric and finds its absent. This should be exceedingly unlikely under most of the write strategies.
If a "live" whisper file is deleted, we'll drop all datapoints and log a dropped create when we attempt to write to it next. If there are further datapoints after this, the metric should be recreated. The old behaviour would have created the file on the write attempt.
We've had this on a moderately sized production system for some time now and haven't spotted any problems. I started with work on #888 by @ploxiln and the original work by @piotr1212, but this approach is slightly different, and happens to have a slightly more readable diff. Differences:
droppedCreate
rather than an exception, I think.There are a few behaviour changes:
Deprecates #888
Fixes https://github.com/graphite-project/graphite-web/issues/629