Backup/Restore of metrics data on new disks

[rickyari]

Just wanted to check on the procedure that should be followed to transfer data of existing metrics to newer disks. Our cluster has 9 nodes and each node has a 2TB dedicated disk attached for storing metrics data. Just wanted to check on the best approach to backup and transfer the data to newer disks.

[rickyari]

@deniszh Please help.

[ritmas]

I have 2 options in my mind:

• use lsyncd to sync files between the disks, e.g.:

  lsyncd -logfile /var/log/lsyncd.log -rsync /disk1/data/ /disk2/data/

• check buckytools functionality to achieve the goal. I haven't tried it for moving data/metrics though

[Civil]

For the bucky tools, I would suggest using go-graphite's fork: https://github.com/go-graphite/buckytools

Instead of the original.

There were some bugfixes and features added.

However about the preferred way, I think people who maintain go-carbon could comment more.

[bom-d-van]

We use buckytools for rebalancing and backfilling.

You need to run buckyd in your go-carbon storage node and can use bucky reblance command for that.

This example assumes you are running a go-carbon cluster is also using hashing jump_fnv1a:

/bin/buckyd -hash jump_fnv1a -node $current_host_ip -prefix /var/lib/carbon/whisper -replicas 1 -sparse $node1_ip $node2_up $current_host_ip $another_node_ip

# This would rebalance the same cluster
/bin/bucky rebalance -f -h target_ip:4242 -w 10 

# This could copy metrics from an existing cluster
/bin/bucky rebalance -f -h target_ip:4242 -w 10 $host_ip_1_in_another_cluster $host_ip_2_in_another_cluster $host_ip_3_in_another_cluster ...

[rickyari]

Thanks for your suggestions