search

grobian / carbon-c-relay

Enhanced C implementation of Carbon relay, aggregator and rewriter
Apache License 2.0
380 stars 107 forks source link

buffering metrics on a disk #271

Closed azhiltsov closed 5 months ago

azhiltsov commented 7 years ago

When relay can't send a metrics out it keeps them in a memory till certain limit. It will be useful to store them on a disk when the memory buffer full. This will give more flexibility for storage reboot/maintenance etc.

grobian commented 7 years ago

Sounds like this would be a bit more thorough rewrite to become memory bound (or a percentage) and try pushing out blocks to disk when exceeding the threshold. It's sort of a 2-stage rocket, where one overflows into the other, which may be too slow to handle, so drop metrics, like the current 1-stage rocket does.

azhiltsov commented 7 years ago

The disk write will be linear, so even spinning disk should handle it fine.

xneo64 commented 7 years ago

@grobian we also encounter this issue frequently. During a cluster member downtime, the relay can't contain all of the pending metrics in memory (even after tuning) - there is simply too much to hold.

Do you know of any approach we may use to counteract this? Could we have the "file" cluster act as a backup of sorts?

grobian commented 7 years ago

carbon-c-relay is more of a router than a cache, a failover target could maybe do what you need.

tantra35 commented 6 years ago

@xneo64 you may use our version of grafsy https://github.com/tantra35/grafsy for this purposes

grobian commented 5 months ago

sounds like there are alternative services that can be used to achieve this purpose