Error in restarting flume for the second ingestion

[hellios78]

I use customeQuery with the INCREMENTAL "$@$".

The running is always OK on the each runcycle of the incremental.

Seems to be a bug when i retart the Flume sevice, raising from the code line https://github.com/keedio/flume-ng-sql-source/blob/develop/src/main/java/org/keedio/flume/source/SQLSourceHelper.java#L268

Should be putting customQuery into QUERY_STATUS_FILE in https://github.com/keedio/flume-ng-sql-source/blob/develop/src/main/java/org/keedio/flume/source/SQLSourceHelper.java#L183

Any suggest of this wrong?

Rgards,

[mvalleavila]

Hello hellios,

Yes you are right, there is a bug whit custom query and writing it in the status file. You catched the code where the custom query and the incremental value should be stored in the json map. I will implement a hotfix and a new release in few days, but feel free to fork and propose a pull request if you want.

Thanks! Marcelo.

[lazaromedina]

Hi, cannot reproduce problem even if flume stops and starts again. Custom query and incremental value (tracking) are already been stored in json.

_agent.sources.sql1.custom.query = SELECT id,firstname FROM customers WHERE id > $@$

1º start flume-sql and load into db 500 rows. 2º load into db 200 rows 3º stop flume and start again 4º load 100 rows

{"Query":"SELECT id,first_name FROM customers WHERE id > $@$","LastIndex":"800","SourceName":"sql1","URL":"jdbc:mysql:\/\/127.0.0.1:3306\/testdb"}

note: SET @@auto_increment_increment=1;