search

grobian / carbon-c-relay

Enhanced C implementation of Carbon relay, aggregator and rewriter
Apache License 2.0
380 stars 107 forks source link

Writing metrics on a disk and then to remote carbon-c-relay (backup) #275

Closed nitr0dv closed 7 years ago

nitr0dv commented 7 years ago

Hello!

I have a task to store metrics (consistantly) on two nodes. I have a node A (MASTER, keepalived) and B (BACKUP, keepalived). Metrics are going to the VIP that points to a MASTER node. And MASTER node redirects them to a BACKUP node. And also all mestrics stores in some file F (file cluster in carbon-c-relay). If I have to do some maintenance on a node B, I turn it off and do this work. So that time metrics are on a node A only and do not rediracted to a node B, because if's off. When works completed, I need to make metrics consistent on both nodes. So I write some BASH script that send metrcis from file F to a node B carbon-c-relay.

Problem is: timestamps of metrics are not aligned. So if I have (in whisper) timestamps 1494852300, 1494852360, 1494852420, ... and want to send metric with timestamp 1494852324 metric will not updated, because (I think) there is no such timestamp in the whisper archives for this metric ;(

Can you help and show me the best way to resolve my problem?

Thanks a lot!

grobian commented 7 years ago

This is a different thing, but why don't you write the metrics to both A and B from the relay? That would remove your problem I think.

nitr0dv commented 7 years ago

Hello! I think not, because if my node B is turned off it doesn't recive new metrics. Just like in my case. So in both cases I need to resend some metrics to the node B.

grobian commented 7 years ago

true, but you can "synchronise" both clusters, look at tools like whisper-fill and the buckytools (https://github.com/jjneely/buckytools) speed-improved version.

nitr0dv commented 7 years ago

So, if you can tell me the alignment function for timestamp I will align metrics' timestamp in the file F and resend them to the node B. I think that alignment function is F(t) = t / interval * interval. No?

grobian commented 7 years ago

Also for that, you can use whisper-fill.py, it also deals with updating possibly overlapping retentions.

Sending your 24 metric should just work, as whisper is automatically assigning the datapoint to a bucket, based on its grid.

nitr0dv commented 7 years ago

But whisper-fill can't do a remote job, no? I'll need to write some wrapper to use it remote.

azhiltsov commented 7 years ago

Buckytools can do. It acts as a client-server, where server exposes REST API to your whisper files.

nitr0dv commented 7 years ago

Thank you all! Will try ;)