try insert large number of Influx series

VictoriaMetrics / vmctl

Victoria Metrics command line tool

36 stars 8 forks source link

try insert large number of Influx series #15

Open barebu opened 4 years ago

barebu commented 4 years ago

Trying to transfer a large data from influx to VM. SHOW SERIES CARDINALITY query returns 21696367. The process lasts a very long time and ends at "fetching series". After 2-3 hours get OOM.

Use ./vmctl influx --influx-addr "http://my_influx:8086" --influx-database db_name --vm-addr "http://vm_insert:8680" --vm-account-id 1 --vm-concurrency 6 --influx-concurrency 6 --influx-chunk-size 20000

hagen1778 commented 4 years ago

Thanks for report! Looks like phase 1 for influx mode should be improved to avoid OOMs. As a workaround try to limit number of series to import at once via --influx-filter-series flag.

valyala commented 3 years ago

FYI, vmctl has been migrated to main VictoriaMetrics repository. New releases for vmctl will be available at vmutils-*.tar.gz package on the releases page starting from v1.53.0 . It is recommended filing new issues and feature requests for vmctl in main VictoriaMetrics repository.