search

keedio / flume-ng-sql-source

Flume Source to import data from SQL Databases
Apache License 2.0
264 stars 164 forks source link

Question about incremental insertion #80

Open qintian95 opened 5 years ago

qintian95 commented 5 years ago

Hello,I have set the query interval but it won't update automatically. I need to manually restart the agent.

test.sources.sqls.type = org.keedio.flume.source.SQLSource

test.sources.sqls.hibernate.connection.url = jdbc:mysql://192.168.199.237:3306/testflume test.sources.sqls.hibernate.connection.user = root test.sources.sqls.hibernate.connection.password = ~!@Mysql@2018 test.sources.sqls.hibernate.connection.autocommit = true test.sources.sqls.hibernate.connection.provider_class = org.hibernate.connection.C3P0ConnectionProvider test.sources.sqls.hibernate.connection.driver_class = com.mysql.jdbc.Driver test.sources.sqls.hibernate.c3p0.min_size=1 test.sources.sqls.hibernate.c3p0.max_size=10 test.sources.sqls.hibernate.dialect = org.hibernate.dialect.MySQL5Dialect

test.sources.sqls.max.rows = 1000000 test.sources.sqls.table = test test.sources.sqls.columns.to.select = * test.sources.sqls.column.name = id
test.sources.sqls.incremental.value = 0 test.sources.sqls.delimiter.entry = , test.sources.sqls.enclose.by.quotes = false test.sources.sqls.run.query.delay=5000 test.sources.sqls.hibernate.dialect = org.hibernate.dialect.MySQL5Dialect

test.sources.sqls.status.file.path = /data/flume test.sources.sqls.status.file.name = sqls.status

test.sources.sqls.channels = c1

test.sinks.hdfssink.channel = c1 test.sinks.hdfssink.type = hdfs test.sinks.hdfssink.hdfs.path = hdfs://nameservice1:8020/flume/mysqls test.sinks.hdfssink.hdfs.fileType = DataStream test.sinks.hdfssink.hdfs.writeFormat = Text test.sinks.hdfssink.hdfs.rollSize = 268435456 test.sinks.hdfssink.hdfs.rollInterval = 0 test.sinks.hdfssink.hdfs.rollCount = 0

test.channels.c1.type = memory

qintian95 commented 5 years ago

hello,I added this property, test.sources.sqls.hibernate.connection.autocommit = true but there have been new problems,the newly inserted line will be read repeatedly

[root@mf01 ~]# hdfs dfs -tail -f /flume/mysqls/FlumeData.1547451242689.tmp 1;haha;13 2;hah;13 3;haa;13 4;ha;13 5;aha;13 6;haa;13 7;qqq;12 8;qdg;12 9;zxcsdf;12 10;zxcsdf;12 11;zxcsdf;12 12;zxcsdf;12 13;zxcsdf;12 14;zxcsdf;12 15;zxcsdf;12 15;zxcsdf;12 15;zxcsdf;12

qintian95 commented 5 years ago

@HbnKing,你好，请问这个问题怎么解决？

HbnKing commented 5 years ago

@qintian95 跟我以前一样的问题呢

qintian95 commented 5 years ago

@HbnKing是咋解决的？😁️

HbnKing commented 5 years ago

@qintian95 自己写了一个啊 ，没办法

qintian95 commented 5 years ago

@HbnKing 你好，方便分享出来吗？这么大的问题作者没发现吗。。。

HbnKing commented 5 years ago

年后吧

qintian95 commented 5 years ago

@HbnKing好的😄

qintian95 commented 5 years ago

@HbnKing 新年快乐，最近方便分享下吗？

HbnKing commented 5 years ago

on my github

shizhenchina commented 5 years ago

hibernate.connection.driver_class参数问题吧，你的配置貌似是使用hibernate自带连接池，我之前没配该参数也一样有问题。 https://www.linuxidc.com/Linux/2014-09/106459.htm