keedio / flume-ng-sql-source

Flume Source to import data from SQL Databases
Apache License 2.0
264 stars 164 forks source link

out of memory : GC overhead exceed error #51

Closed vicky1728 closed 6 years ago

vicky1728 commented 6 years ago

using jar version 1.4.1 and getting this error image

i have increased the Xms to 8gb and Xmx to 16gb still getting the error please help

randidwiputra commented 6 years ago

in your file configuration (flume-env.sh). try with configuration below JAVA_OPTS="-Xms1000m -Xmx1000m -Xss256k -XX:MaxDirectMemorySize=256m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC"

your linux systems running in virtual machine ? Your memory is not enough maybe, so outofmemoryerror

vicky1728 commented 6 years ago

Thank you for the reply sir but the problem still persists (same problem) Linux is in VM. If I'm running jar version 1.3.7 then it gives me error of connection password not set and for version 1.4.1 GC overhead error. Please help

lazaromedina commented 6 years ago

Hi vicky1728, please update to latest release of flume-sql, or at least use 1.4.3(compatible with Apache Flume prior to 1.7.0)

best, Luis

vicky1728 commented 6 years ago

Hi Luis I will drop all the screenshots today please see. thanks

vicky1728 commented 6 years ago

image image

this is error i'm getting on another cluster

vicky1728 commented 6 years ago

untitled this is my script

lazaromedina commented 6 years ago

Hi vick1728, please check for driver for mysql connection, download and deploy.

Luis.

vicky1728 commented 6 years ago

Hello Luis image i have all the drivers already sir Vicky

vicky1728 commented 6 years ago

image this error I'm getting using version 1.3.7

lazaromedina commented 6 years ago

Hi vicky1728, could you please follow these steps:

best, luis

vicky1728 commented 6 years ago

flume-ng agent --conf ./conf -f /usr/hdp/current/flume-server/conf/agent/flume.conf -n agent -Dflume.root.logger=INFO,console Info: Including Hadoop libraries found via (/bin/hadoop) for HDFS access Info: Excluding /usr/hdp/2.6.3.0-235/hadoop/lib/slf4j-api-1.7.10.jar from classpath Info: Excluding /usr/hdp/2.6.3.0-235/hadoop/lib/slf4j-log4j12-1.7.10.jar from classpath Info: Excluding /usr/hdp/2.6.3.0-235/tez/lib/slf4j-api-1.7.5.jar from classpath Info: Including HBASE libraries found via (/bin/hbase) for HBASE access Info: Excluding /usr/hdp/2.6.3.0-235/hbase/lib/slf4j-api-1.7.7.jar from classpath Info: Excluding /usr/hdp/2.6.3.0-235/hadoop/lib/slf4j-api-1.7.10.jar from classpath Info: Excluding /usr/hdp/2.6.3.0-235/hadoop/lib/slf4j-log4j12-1.7.10.jar from classpath Info: Excluding /usr/hdp/2.6.3.0-235/tez/lib/slf4j-api-1.7.5.jar from classpath Info: Excluding /usr/hdp/2.6.3.0-235/hadoop/lib/slf4j-api-1.7.10.jar from classpath Info: Excluding /usr/hdp/2.6.3.0-235/hadoop/lib/slf4j-log4j12-1.7.10.jar from classpath Info: Including Hive libraries found via () for Hive access

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. at sun.reflect.GeneratedConstructorAccessor6.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1121) at com.mysql.jdbc.MysqlIO.(MysqlIO.java:357) at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2484) at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2521) at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2306) at com.mysql.jdbc.ConnectionImpl.(ConnectionImpl.java:839) at com.mysql.jdbc.JDBC4Connection.(JDBC4Connection.java:49) at sun.reflect.GeneratedConstructorAccessor4.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:421) at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:350) at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:146) at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:195) at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:184) at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:200) at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1086) at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1073) at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:44) at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1810) at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:648) Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at java.net.Socket.connect(Socket.java:538) at java.net.Socket.(Socket.java:434) at java.net.Socket.(Socket.java:244) at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:259) at com.mysql.jdbc.MysqlIO.(MysqlIO.java:307) ... 20 more 18/02/09 11:44:20 WARN resourcepool.BasicResourcePool: com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@17fef245 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. at sun.reflect.GeneratedConstructorAccessor6.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1121) at com.mysql.jdbc.MysqlIO.(MysqlIO.java:357) at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2484) at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2521) at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2306) at com.mysql.jdbc.ConnectionImpl.(ConnectionImpl.java:839) at com.mysql.jdbc.JDBC4Connection.(JDBC4Connection.java:49) at sun.reflect.GeneratedConstructorAccessor4.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:421) at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:350) at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:146) at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:195) at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:184) at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:200) at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1086) at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1073) at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:44) at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1810) at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:648) Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at java.net.Socket.connect(Socket.java:538) at java.net.Socket.(Socket.java:434) at java.net.Socket.(Socket.java:244) at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:259) at com.mysql.jdbc.MysqlIO.(MysqlIO.java:307) ... 20 more 18/02/09 11:44:20 WARN resourcepool.BasicResourcePool: com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@188bacb -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. at sun.reflect.GeneratedConstructorAccessor6.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1121) at com.mysql.jdbc.MysqlIO.(MysqlIO.java:357) at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2484) at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2521) at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2306) at com.mysql.jdbc.ConnectionImpl.(ConnectionImpl.java:839) at com.mysql.jdbc.JDBC4Connection.(JDBC4Connection.java:49) at sun.reflect.GeneratedConstructorAccessor4.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:421) at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:350) at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:146) at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:195) at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:184) at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:200) at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1086) at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1073) at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:44) at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1810) at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:648) Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at java.net.Socket.connect(Socket.java:538) at java.net.Socket.(Socket.java:434) at java.net.Socket.(Socket.java:244) at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:259) at com.mysql.jdbc.MysqlIO.(MysqlIO.java:307) ... 20 more 18/02/09 11:44:20 WARN resourcepool.BasicResourcePool: Having failed to acquire a resource, com.mchange.v2.resourcepool.BasicResourcePool@195a360a is interrupting all Threads waiting on a resource to check out. Will try again in response to new client requests. 18/02/09 11:44:20 WARN resourcepool.BasicResourcePool: Having failed to acquire a resource, com.mchange.v2.resourcepool.BasicResourcePool@195a360a is interrupting all Threads waiting on a resource to check out. Will try again in response to new client requests. 18/02/09 11:44:20 WARN resourcepool.BasicResourcePool: Having failed to acquire a resource, com.mchange.v2.resourcepool.BasicResourcePool@195a360a is interrupting all Threads waiting on a resource to check out. Will try again in response to new client requests. 18/02/09 11:44:20 WARN internal.JdbcServicesImpl: HHH000342: Could not obtain connection to query metadata : Connections could not be acquired from the underlying database! 18/02/09 11:44:20 ERROR node.AbstractConfigurationProvider: Source sql-source has been removed due to an error during configuration org.hibernate.HibernateException: Access to DialectResolutionInfo cannot be null when 'hibernate.dialect' not set at org.hibernate.engine.jdbc.dialect.internal.DialectFactoryImpl.determineDialect(DialectFactoryImpl.java:104) at org.hibernate.engine.jdbc.dialect.internal.DialectFactoryImpl.buildDialect(DialectFactoryImpl.java:71) at org.hibernate.engine.jdbc.internal.JdbcServicesImpl.configure(JdbcServicesImpl.java:205) at org.hibernate.boot.registry.internal.StandardServiceRegistryImpl.configureService(StandardServiceRegistryImpl.java:111) at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:234) at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:206) at org.hibernate.cfg.Configuration.buildTypeRegistrations(Configuration.java:1887) at org.hibernate.cfg.Configuration.buildSessionFactory(Configuration.java:1845) at org.keedio.flume.source.HibernateHelper.establishSession(HibernateHelper.java:67) at org.keedio.flume.source.SQLSource.configure(SQLSource.java:73) at org.apache.flume.conf.Configurables.configure(Configurables.java:41) at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:331) at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:102) at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 18/02/09 11:44:20 INFO sink.DefaultSinkFactory: Creating instance of sink: store, type: hdfs 18/02/09 11:44:20 INFO hdfs.HDFSEventSink: Hadoop Security enabled: false 18/02/09 11:44:20 INFO node.AbstractConfigurationProvider: Channel newmem connected to [store] 18/02/09 11:44:20 INFO node.Application: Starting new configuration:{ sourceRunners:{} sinkRunners:{store=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@3bc651e1 counterGroup:{ name:null counters:{} } }} channels:{newmem=org.apache.flume.channel.MemoryChannel{name: newmem}} } 18/02/09 11:44:20 INFO node.Application: Starting Channel newmem 18/02/09 11:44:20 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: newmem: Successfully registered new MBean. 18/02/09 11:44:20 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: newmem started 18/02/09 11:44:20 INFO node.Application: Starting Sink store 18/02/09 11:44:20 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SINK, name: store: Successfully registered new MBean. 18/02/09 11:44:20 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: store started ^C18/02/09 11:45:04 INFO lifecycle.LifecycleSupervisor: Stopping lifecycle supervisor 11 18/02/09 11:45:04 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider stopping 18/02/09 11:45:04 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: newmem stopped 18/02/09 11:45:04 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: CHANNEL, name: newmem. channel.start.time == 1518156860545 18/02/09 11:45:04 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: CHANNEL, name: newmem. channel.stop.time == 1518156904495 18/02/09 11:45:04 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: CHANNEL, name: newmem. channel.capacity == 1000 18/02/09 11:45:04 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: CHANNEL, name: newmem. channel.current.size == 0 18/02/09 11:45:04 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: CHANNEL, name: newmem. channel.event.put.attempt == 0 18/02/09 11:45:04 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: CHANNEL, name: newmem. channel.event.put.success == 0 18/02/09 11:45:04 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: CHANNEL, name: newmem. channel.event.take.attempt == 7 18/02/09 11:45:04 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: CHANNEL, name: newmem. channel.event.take.success == 0 18/02/09 11:45:04 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: store stopped 18/02/09 11:45:04 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: store. sink.start.time == 1518156860547 18/02/09 11:45:04 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: store. sink.stop.time == 1518156904498 18/02/09 11:45:04 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: store. sink.batch.complete == 0 18/02/09 11:45:04 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: store. sink.batch.empty == 7 18/02/09 11:45:04 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: store. sink.batch.underflow == 0 18/02/09 11:45:04 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: store. sink.connection.closed.count == 0 18/02/09 11:45:04 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: store. sink.connection.creation.count == 0 18/02/09 11:45:04 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: store. sink.connection.failed.count == 0 18/02/09 11:45:04 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: store. sink.event.drain.attempt == 0 18/02/09 11:45:04 INFO instrumentation.MonitoredCounterGroup: Shutdown Metric for type: SINK, name: store. sink.event.drain.sucess == 0

vicky1728 commented 6 years ago

Sir Luis, above is the error I'm getting after doing all the process you told me to do

Regards Vicky

vicky1728 commented 6 years ago

here is my script of flume

agent.channels=newmem agent.channels.newmem.capacity=1000 agent.channels.newmem.jdbc.connection.url=jdbc:mysql://10.84.204.222:3306/cctns agent.channels.newmem.jdbc.hibernate.connection.driver_class=com.mysql.jdbc.Driver agent.channels.newmem.type=memory agent.channels.newmem.use-fast-replay=TRUE agent.sinks=store agent.sinks.store.channel=newmem agent.sinks.store.hdfs.file.type=DataStream agent.sinks.store.hdfs.path=hdfs://nichadoop1.nic.in:8020/root agent.sinks.store.type=hdfs agent.sources=sql-source agent.sources.sql-source.channels=newmem agent.sources.sql-source.database=cctns_ram agent.sources.sql-source.hibernate.connection.autocommit=true agent.sources.sql-source.hibernate.connection.password=root agent.sources.sql-source.hibernate.connection.url=jdbc:mysql://10.84.204.222:3306/cctns agent.sources.sql-source.hibernate.connection.user=root agent.sources.sql-source.run.querry.delay=1000 agent.sources.sql-source.selector.type=replicating agent.sources.sql-source.status.file.name=sql-source.status agent.sources.sql-source.status.file.path=/usr/lib/flume agent.sources.sql-source.table=t_fir_registration agent.sources.sql-source.type=org.keedio.flume.source.SQLSource agent.sources.sql-source.hibernate.connection.dialect=org.hibernate.dialect.MySQL5Dialect agent.sources.sql-source.hibernate.connection.provider_class=org.hibernate.connection.C3P0ConnectionProvider agent.sources.sql-source.hibernate.C3P0.min_size=1 agent.sources.sql-source.hibernate.C3P0.max_size=10

Please help

thanks and regards Vicky

lazaromedina commented 6 years ago

Hi vicky1728, if you check you log, there seems to be a problem with the connection to the database mysql, not with flume source component: ...... (30). Last acquisition attempt exception: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. ....... Caused by: java.net.ConnectException: Connection refused (Connection refused) ....... The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. .......

you must check connection to database first, user, password, is server up? My recommendation is that check first link to database. If link is ok, try a small and simple configuration for flume-sql. And after that move your config to your case, it will be much easier to debug. Please, I kindly ask you to understand, that the open issue started with a memory problem and now seems a comunication problem for which I can not give you support because it is not related to flume-sql-component.

best, Luis

vicky1728 commented 6 years ago

Hi sir I will check to that and one more error and it says

"ERROR node.AbstractConfigurationProvider: Source sql-source has been removed due to an error during configuration org.hibernate.HibernateException: Access to DialectResolutionInfo cannot be null when 'hibernate.dialect' not set at org.hibernate.engine.jdbc.dialect.internal.DialectFactoryImpl.determineDialect(DialectFactoryImpl.java:104)"

I have already given this property in my script agent.sources.sql-source.hibernate.connection.dialect=org.hibernate.dialect.MySQL5Dialect

still?

Thanks Vicky