[Q] Best practice to show the missing data in graphite/grafana.

[veramasu]

Hi Team,

We have the metrics solution implemented with graphite and rendered using grafana dashboards. We have 2 nodes acting as Active and Standby. Both the nodes gets the metrics from collectd and stored in the carbon whisper db. We have a load balance node (Haproxy) which sends the http request to always node1. When node1 is down, the request will be directed to node2.

Now the situation is, node1 is taken down for maintenance, so all the grafana requests are served by node2, so user dont see any gap in the grafana. This works great.

But when node1 is comes back online, user see a gap in the grafana, as node1 didnt have the entry for the time it was down.

the requirement is to show the missing data and there are few options we are thinking of

1) run a rsync and get the wsp files from the standby server which has the data during node1 was down. 2) get the wsp files from node2 and run whisper-fill to fill only the missing data.

Both the above approaches takes considerable amount of time for syncing so, we kept these as backup options. Option 3 when we looked into the graphite-web/local_settings.py there are few settings which gets the missing data from the REMOTE server. CLUSTER_SERVERS ---> i set both the local and remote server. But its still not doing any query to the remote and fill the gaps. REMOTE_RENDERING = True RENDERING_HOSTS --> both the server

Question? Let me know what we miss here. As i said there is no replication from node1 to node2. Both acts as individual copy. Option 4

In the grafana we added both the node as datasource. in the Panel we use mixed and its showing the data from both the nodes. But we are not able to use any transform,aggregate functions with that. Can you please share some example how to write a graphite query from mixed datasurce but merge them into a single graph.

for example here is the query groupByNode(com.node.vm01.cpu.idle, 4, ‘sum’) from DataSource 1 groupByNode(com.node.vm01.cpu.idle, 4, ‘sum’) from DataSource 2

when it renders in graana it shows both the output as expected. But the legends shows twice as they are 2 different queries . want to merge both of them into single. for example it returns for cpu-0 twice and i want to merge them both to single cpu-0.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[deniszh]

@veramasu : sorry, missed your question somehow. Unfortunately, anything except Option 1 will not work. You can use rsync or bucky-fill to sync data.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.