search

timescale / outflux

Export data from InfluxDB to TimescaleDB
Apache License 2.0
90 stars 22 forks source link

Dot (.) in InfluxDB measurement name is interpreted wrong #56

Closed erlfos closed 5 years ago

erlfos commented 5 years ago

Hi

I am testing Outflux to see if Timescale can offer an alternative to InfluxDB, but our measurements in InfluxDB often have dots in the measurement name (example: VS.AsrRxMalformedPkts) and this causes Outflux to fail. Outflux translates "VS" as the retention policy and "AsrRxMalformedPkts" as the measurement name, and the resulting query fails:

2019/07/01 08:39:18 Selected input database: spgw
2019/07/01 08:39:18 Overriding PG environment variables for connection with: dbname=spgw user=postgres
2019/07/01 08:39:18 pipe_VS.AsrRxMalformedPkts starting execution
2019/07/01 08:39:18 Discovering influx schema for measurement: VS.AsrRxMalformedPkts
2019/07/01 08:39:19 could not transfer schema for measurement 'VS.AsrRxMalformedPkts'
pipe_VS.AsrRxMalformedPkts: could not prepare extractor
pipe_VS.AsrRxMalformedPkts_ext: could not fetch data set definition for measure: VS.AsrRxMalformedPkts
could not discover the fields of measure 'VS.AsrRxMalformedPkts'
error fetching fields for measurement 'VS.AsrRxMalformedPkts'
error executing query: SHOW FIELD KEYS FROM "VS"."AsrRxMalformedPkts"
retention policy not found: VS

Here is the output from Influx:

> SHOW FIELD KEYS FROM "VS"."AsrRxMalformedPkts"
ERR: retention policy not found: VS
> SHOW FIELD KEYS FROM "VS.AsrRxMalformedPkts"
name: VS.AsrRxMalformedPkts
fieldKey fieldType
-------- ---------
value    float
>

If there is a dot in the measurement name the query will have to be like this: SHOW FIELD KEYS FROM "VS.AsrRxMalformedPkts"

atanasovskib commented 5 years ago

Thank you for trying out Outflux. Can you show us how you passed the flags when starting the tool? (No need for connection stuff, just measurement name and other options) :)

erlfos commented 5 years ago

I run Outflux in a TimescaleDB docker container with this command: bash-4.4# ./outflux schema-transfer --input-server http://influxdb:8086 spgw --output-conn="dbname=spgw user=postgres"

atanasovskib commented 5 years ago

Confirmed, there is trouble when having dots in the measure name. You can sidestep it by using quotes and adding the autogen retention policy, but TimescaleDB/Postgres ends up trying to create a schema with the first part of the measure name (this case schema VS, table AsrRxMalformedPkts). Will issue a bugfix and then do another release. Thank you for reporting it

atanasovskib commented 5 years ago

@erlfos We have a fix for this issue in #60 . In the same PR we add explicit support for retention policies, but we also fixed the bug where the dot in the measurement name is interpreted as a RP. You can try out the code from the PR to confirm that it's been fixed. After our checks pass we'll merge it in master and in a few days issue a new release