search

timescale / outflux

Export data from InfluxDB to TimescaleDB
Apache License 2.0
86 stars 22 forks source link

Outflux fails silently when Influxdb server has query-timeout defined #91

Open wiebeytec opened 2 years ago

wiebeytec commented 2 years ago

Running outflux, it consistently stops after about 900 seconds:

2021/12/08 16:06:35 pipe_479_ext: Extracted 83655000 rows from Influx
2021/12/08 16:06:35 pipe_479_ext: Extracted 83670000 rows from Influx
2021/12/08 16:06:35 pipe_479_ing: Complete. Inserted 83670000 rows.
2021/12/08 16:06:35 All pipelines finished
2021/12/08 16:06:35 Migration execution time: 940.285 seconds

But, it's far from complete. It managed to load about a day of data, but it should load 6 months.

Our Influx server has a query-timeout of 900 seconds, so this seems to be because of that.

I can't even work around it by batching using multiple calls to outflux, because only DropAndCreate works, because of querying the Timescale version 2 schema fails.

Expected/requested behavior:

  1. At least detection of the error. According to the documentation, Influx 1 should say: ERR: query timeout reached.
  2. Ability to batch the reading in x number of rows, or time range, etc.

Outflux version 0.2.0