Open FFYYang opened 6 years ago
FFYYang
commented
6 years ago I wrongly spelled the source name....... agent.sources.sqlSource.hibernate.connection.autocommit = true the correct should be : agent.sources.sql-source.hibernate.connection.autocommit = true
lazaromedina
commented
5 years ago Hi YasinQiu, i cannot see error on you log, but in your config you are using custom query property. Please can you try in your environment flume-sql-source-1.5.2-SNAPSHOT or wait until next release with a fix for a bug-#57 that may be the problem your report.
best
Hi, I'm trying to stream data from mysql table into logger. But I can't stream the newly inserted data in mysql table.
I'm using flume-release-1.8.0, flume-ng-sql-source-1.5.0
My flume.conf looks like this, agent.sources = sql-source agent.channels = ch1 agent.sinks = k1
agent.sources = sql-source agent.channels = ch1 agent.sinks = k1
agent.sources.sql-source.type = org.keedio.flume.source.SQLSource
agent.sources.sql-source.hibernate.connection.url = jdbc:mysql://localhost:3306/test1 agent.sources.sql-source.hibernate.connection.user = root agent.sources.sql-source.hibernate.connection.password = 123456 agent.sources.sqlSource.hibernate.connection.autocommit = true agent.sources.sql-source.table = stu
agent.sources.sql-source.run.query.delay = 2000 agent.sources.sql-source.status.file.path = /home/yasinqiu/flume/mysql-test agent.sources.sql-source.status.file.name = sql-source.status
agent.sources.sql-source.custom.query = SELECT * FROM stu WHERE id > $@$
agent.sources.sql-source.hibernate.dialect = org.hibernate.dialect.MySQL5Dialect agent.sources.sql-source.hibernate.connection.driver_class = com.mysql.jdbc.Driver
agent.sinks.k1.type = logger
agent.channels.ch1.type = memory agent.channels.ch1.capacity = 1000 agent.channels.ch1.transactionCapacity = 100
agent.sources.sql-source.channels = ch1 agent.sinks.k1.channel = ch1
================================================= when i insert a new row in mysql table, i cannot see the new inserted data from logger.
the console's information is shown below: Info: Sourcing environment configuration script /home/yasinqiu/flume/apache-flume-1.8.0-bin/conf/flume-env.sh Info: Including Hive libraries found via () for Hive access
com.mysql.jdbc.Driver'. This is deprecated. The new driver class iscom.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary. 2018-07-08 15:32:30,652 (conf-file-poller-0) [INFO - org.hibernate.engine.jdbc.connections.internal.DriverManagerConnectionProviderImpl.buildCreator(DriverManagerConnectionProviderImpl.java:166)] HHH000401: using driver [com.mysql.jdbc.Driver] at URL [jdbc:mysql://localhost:3306/test1] 2018-07-08 15:32:30,653 (conf-file-poller-0) [INFO - org.hibernate.engine.jdbc.connections.internal.DriverManagerConnectionProviderImpl.buildCreator(DriverManagerConnectionProviderImpl.java:175)] HHH000046: Connection properties: {user=root, password=****} 2018-07-08 15:32:30,653 (conf-file-poller-0) [INFO - org.hibernate.engine.jdbc.connections.internal.DriverManagerConnectionProviderImpl.buildCreator(DriverManagerConnectionProviderImpl.java:180)] HHH000006: Autocommit mode: false 2018-07-08 15:32:30,654 (conf-file-poller-0) [INFO - org.hibernate.engine.jdbc.connections.internal.DriverManagerConnectionProviderImpl.configure(DriverManagerConnectionProviderImpl.java:102)] HHH000115: Hibernate connection pool size: 20 (min=1) 2018-07-08 15:32:30,948 (conf-file-poller-0) [INFO - org.hibernate.dialect.Dialect.