Closed vicky1728 closed 6 years ago
in your file configuration (flume-env.sh). try with configuration below
JAVA_OPTS="-Xms1000m -Xmx1000m -Xss256k -XX:MaxDirectMemorySize=256m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC"
your linux systems running in virtual machine ? Your memory is not enough maybe, so outofmemoryerror
Thank you for the reply sir but the problem still persists (same problem) Linux is in VM. If I'm running jar version 1.3.7 then it gives me error of connection password not set and for version 1.4.1 GC overhead error. Please help
Hi vicky1728, please update to latest release of flume-sql, or at least use 1.4.3(compatible with Apache Flume prior to 1.7.0)
best, Luis
Hi Luis I will drop all the screenshots today please see. thanks
this is error i'm getting on another cluster
this is my script
Hi vick1728, please check for driver for mysql connection, download and deploy.
Luis.
Hello Luis i have all the drivers already sir Vicky
this error I'm getting using version 1.3.7
Hi vicky1728, could you please follow these steps:
best, luis
flume-ng agent --conf ./conf -f /usr/hdp/current/flume-server/conf/agent/flume.conf -n agent -Dflume.root.logger=INFO,console Info: Including Hadoop libraries found via (/bin/hadoop) for HDFS access Info: Excluding /usr/hdp/2.6.3.0-235/hadoop/lib/slf4j-api-1.7.10.jar from classpath Info: Excluding /usr/hdp/2.6.3.0-235/hadoop/lib/slf4j-log4j12-1.7.10.jar from classpath Info: Excluding /usr/hdp/2.6.3.0-235/tez/lib/slf4j-api-1.7.5.jar from classpath Info: Including HBASE libraries found via (/bin/hbase) for HBASE access Info: Excluding /usr/hdp/2.6.3.0-235/hbase/lib/slf4j-api-1.7.7.jar from classpath Info: Excluding /usr/hdp/2.6.3.0-235/hadoop/lib/slf4j-api-1.7.10.jar from classpath Info: Excluding /usr/hdp/2.6.3.0-235/hadoop/lib/slf4j-log4j12-1.7.10.jar from classpath Info: Excluding /usr/hdp/2.6.3.0-235/tez/lib/slf4j-api-1.7.5.jar from classpath Info: Excluding /usr/hdp/2.6.3.0-235/hadoop/lib/slf4j-api-1.7.10.jar from classpath Info: Excluding /usr/hdp/2.6.3.0-235/hadoop/lib/slf4j-log4j12-1.7.10.jar from classpath Info: Including Hive libraries found via () for Hive access
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.GeneratedConstructorAccessor6.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1121)
at com.mysql.jdbc.MysqlIO.
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.GeneratedConstructorAccessor6.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1121)
at com.mysql.jdbc.MysqlIO.
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.GeneratedConstructorAccessor6.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1121)
at com.mysql.jdbc.MysqlIO.
Sir Luis, above is the error I'm getting after doing all the process you told me to do
Regards Vicky
here is my script of flume
agent.channels=newmem agent.channels.newmem.capacity=1000 agent.channels.newmem.jdbc.connection.url=jdbc:mysql://10.84.204.222:3306/cctns agent.channels.newmem.jdbc.hibernate.connection.driver_class=com.mysql.jdbc.Driver agent.channels.newmem.type=memory agent.channels.newmem.use-fast-replay=TRUE agent.sinks=store agent.sinks.store.channel=newmem agent.sinks.store.hdfs.file.type=DataStream agent.sinks.store.hdfs.path=hdfs://nichadoop1.nic.in:8020/root agent.sinks.store.type=hdfs agent.sources=sql-source agent.sources.sql-source.channels=newmem agent.sources.sql-source.database=cctns_ram agent.sources.sql-source.hibernate.connection.autocommit=true agent.sources.sql-source.hibernate.connection.password=root agent.sources.sql-source.hibernate.connection.url=jdbc:mysql://10.84.204.222:3306/cctns agent.sources.sql-source.hibernate.connection.user=root agent.sources.sql-source.run.querry.delay=1000 agent.sources.sql-source.selector.type=replicating agent.sources.sql-source.status.file.name=sql-source.status agent.sources.sql-source.status.file.path=/usr/lib/flume agent.sources.sql-source.table=t_fir_registration agent.sources.sql-source.type=org.keedio.flume.source.SQLSource agent.sources.sql-source.hibernate.connection.dialect=org.hibernate.dialect.MySQL5Dialect agent.sources.sql-source.hibernate.connection.provider_class=org.hibernate.connection.C3P0ConnectionProvider agent.sources.sql-source.hibernate.C3P0.min_size=1 agent.sources.sql-source.hibernate.C3P0.max_size=10
Please help
thanks and regards Vicky
Hi vicky1728, if you check you log, there seems to be a problem with the connection to the database mysql, not with flume source component: ...... (30). Last acquisition attempt exception: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. ....... Caused by: java.net.ConnectException: Connection refused (Connection refused) ....... The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. .......
you must check connection to database first, user, password, is server up? My recommendation is that check first link to database. If link is ok, try a small and simple configuration for flume-sql. And after that move your config to your case, it will be much easier to debug. Please, I kindly ask you to understand, that the open issue started with a memory problem and now seems a comunication problem for which I can not give you support because it is not related to flume-sql-component.
best, Luis
Hi sir I will check to that and one more error and it says
"ERROR node.AbstractConfigurationProvider: Source sql-source has been removed due to an error during configuration org.hibernate.HibernateException: Access to DialectResolutionInfo cannot be null when 'hibernate.dialect' not set at org.hibernate.engine.jdbc.dialect.internal.DialectFactoryImpl.determineDialect(DialectFactoryImpl.java:104)"
I have already given this property in my script agent.sources.sql-source.hibernate.connection.dialect=org.hibernate.dialect.MySQL5Dialect
still?
Thanks Vicky
using jar version 1.4.1 and getting this error
i have increased the Xms to 8gb and Xmx to 16gb still getting the error please help