Closed valyala closed 4 years ago
It is better to use native format for data import/export, since it provides up to 50x faster performance than JSON line format and up to 100x lower disk space requirements. Exporting and importing in native format has been added in VictoriaMetrics v1.42.0. See more details in the following docs:
@valyala feature for migrating data between VM instances was added in https://github.com/VictoriaMetrics/vmctl/releases/tag/0.3.0
Sometimes it is necessary to migrate data between VictoriaMetrics instances. For example, when migrating data from single-node version to cluster version of VictoriaMetrics. Or when migrating data from a single cluster to another cluster.
So let's add support for data migration between VictoriaMetrics instances into
vmctl
tool.High level overview of the task:
vmctl
should accept label filters and time range for metrics to migrate. For example-label-filters={foo="bar",baz=~"aaa.*"} -start=2020-07-01T00:00:00Z -end=2020-08-01T00:00:00Z
.vmctl
must read the requested data from the configured source via /api/v1/export and ingest the data to the configured destination via /api/v1/import.The
/api/v1/export
may return big amounts of data, so it would be great ifvmctl
could stream data between source and destination without the need to read all the data from the source before writing it to the destination.