search

vimeo / whisper-to-influxdb

migrate (import) graphite data from whisper to influxdb
Apache License 2.0
58 stars 13 forks source link

add path to finishedfiles when skip whisperfile #2

Closed liyichao closed 7 years ago

liyichao commented 9 years ago
  1. Think this scenario, we ignore whisper error and do not ignore influxdb error, then there is a wsp file A of size 0, it is ignored and processing continues, however, the firstInProcess is always A, even though we may process many files beyond. And when influxdb error happens, the program exists, the next file to continue is A, so we stuck at A.
  2. When influxdb error happens, it may be data value error ( I encountered value +Inf, which was read from a whisper file) and it is useless to submit to influxdb after 5 minutes, a better option will be to record the error path and continue to the next series, user can manually handle these files later.
Dieterbe commented 9 years ago
  1. you're right. we should mark it as "finished" so that the firstinProgress updates correctly
  2. with your patch, we don't even need for loop anymore. but i think we should keep the for loop and if influxdb tells is what kind of error (bad value from whisper -> break), if influxdb-native error, retry later. or if it doesn't tell what the error was, maybe retry a few times and then give up.

also we could just avoid this case all together: if we read a whisper file and it says +inf and we know influxdb won't support it, we can treat it as a bad value and filter out the bad data so that we only submit good values into influx so that any influx errors can be retried later.

claytondukes commented 8 years ago

Any chance someone can update this for influx v2?