VictoriaMetrics / vmctl

Victoria Metrics command line tool
36 stars 8 forks source link

Add support for data migration between VictoriaMetrics instances #21

Closed valyala closed 4 years ago

valyala commented 4 years ago

Sometimes it is necessary to migrate data between VictoriaMetrics instances. For example, when migrating data from single-node version to cluster version of VictoriaMetrics. Or when migrating data from a single cluster to another cluster.

So let's add support for data migration between VictoriaMetrics instances into vmctl tool.

High level overview of the task:

The /api/v1/export may return big amounts of data, so it would be great if vmctl could stream data between source and destination without the need to read all the data from the source before writing it to the destination.

valyala commented 4 years ago

It is better to use native format for data import/export, since it provides up to 50x faster performance than JSON line format and up to 100x lower disk space requirements. Exporting and importing in native format has been added in VictoriaMetrics v1.42.0. See more details in the following docs:

hagen1778 commented 4 years ago

@valyala feature for migrating data between VM instances was added in https://github.com/VictoriaMetrics/vmctl/releases/tag/0.3.0