search

keedio / flume-ng-sql-source

Flume Source to import data from SQL Databases
Apache License 2.0
264 stars 164 forks source link

Exception in thread "PollableSourceRunner-SQLSource-s1" java.lang.OutOfMemoryError: Java heap space #40

Closed starbow727 closed 7 years ago

starbow727 commented 7 years ago

2017-09-05 11:42:47,813 (lifecycleSupervisor-1-0) [INFO - org.apache.kafka.common.config.AbstractConfig.logAll(AbstractConfig.java:171)] ProducerConfig values: request.timeout.ms = 30000 retry.backoff.ms = 100 buffer.memory = 33554432 ssl.truststore.password = null batch.size = 16384 ssl.keymanager.algorithm = SunX509 receive.buffer.bytes = 32768 ssl.cipher.suites = null ssl.key.password = null sasl.kerberos.ticket.renew.jitter = 0.05 ssl.provider = null sasl.kerberos.service.name = null max.in.flight.requests.per.connection = 5 sasl.kerberos.ticket.renew.window.factor = 0.8 bootstrap.servers = [sdzn-cdh03:9092] client.id = max.request.size = 1048576 acks = 1 linger.ms = 0 sasl.kerberos.kinit.cmd = /usr/bin/kinit ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1] metadata.fetch.timeout.ms = 60000 ssl.endpoint.identification.algorithm = null ssl.keystore.location = null value.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer ssl.truststore.location = null ssl.keystore.password = null key.serializer = class org.apache.kafka.common.serialization.StringSerializer block.on.buffer.full = false metrics.sample.window.ms = 30000 metadata.max.age.ms = 300000 security.protocol = PLAINTEXT ssl.protocol = TLS sasl.kerberos.min.time.before.relogin = 60000 timeout.ms = 30000 connections.max.idle.ms = 540000 ssl.trustmanager.algorithm = PKIX metric.reporters = [] compression.type = none ssl.truststore.type = JKS max.block.ms = 60000 retries = 0 send.buffer.bytes = 131072 partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner reconnect.backoff.ms = 50 metrics.num.samples = 2 ssl.keystore.type = JKS

2017-09-05 11:42:47,867 (lifecycleSupervisor-1-0) [INFO - org.apache.kafka.common.utils.AppInfoParser$AppInfo.(AppInfoParser.java:82)] Kafka version : 0.9.0-kafka-2.0.0 2017-09-05 11:42:47,868 (lifecycleSupervisor-1-0) [INFO - org.apache.kafka.common.utils.AppInfoParser$AppInfo.(AppInfoParser.java:83)] Kafka commitId : unknown 2017-09-05 11:42:47,869 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:120)] Monitored counter group for type: SINK, name: k1: Successfully registered new MBean. 2017-09-05 11:42:47,869 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:96)] Component type: SINK, name: k1 started Exception in thread "PollableSourceRunner-SQLSource-s1" java.lang.OutOfMemoryError: Java heap space at oracle.jdbc.driver.PhysicalConnection.getCharBuffer(PhysicalConnection.java:7018) at oracle.jdbc.driver.OracleStatement.prepareAccessors(OracleStatement.java:907) at oracle.jdbc.driver.T4CTTIdcb.receiveCommon(T4CTTIdcb.java:261) at oracle.jdbc.driver.T4CTTIdcb.receive(T4CTTIdcb.java:127) at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:992) at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:194) at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:791) at oracle.jdbc.driver.T4CPreparedStatement.executeMaybeDescribe(T4CPreparedStatement.java:866) at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1186) at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3387) at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3431) at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1491) at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:116) at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:82) at org.hibernate.loader.Loader.getResultSet(Loader.java:2066) at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1863) at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1839) at org.hibernate.loader.Loader.doQuery(Loader.java:910) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:355) at org.hibernate.loader.Loader.doList(Loader.java:2554) at org.hibernate.loader.Loader.doList(Loader.java:2540) at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2370) at org.hibernate.loader.Loader.list(Loader.java:2365) at org.hibernate.loader.custom.CustomLoader.list(CustomLoader.java:353) at org.hibernate.internal.SessionImpl.listCustomQuery(SessionImpl.java:1909) at org.hibernate.internal.AbstractSessionImpl.list(AbstractSessionImpl.java:311) at org.hibernate.internal.SQLQueryImpl.list(SQLQueryImpl.java:141) at org.keedio.flume.source.HibernateHelper.executeQuery(HibernateHelper.java:122) at org.keedio.flume.source.SQLSource.process(SQLSource.java:89) at org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:137) at java.lang.Thread.run(Thread.java:745)

mvalleavila commented 7 years ago

It seems that your Flume JVM is started with not enough heap space. Read about Xmx java property https://docs.oracle.com/cd/E13150_01/jrockit_jvm/jrockit/jrdocs/refman/optionX.html

And use it in flume-env.sh file, then restart your agent. https://cwiki.apache.org/confluence/display/FLUME/Flume%27s+Memory+Consumption

starbow727 commented 7 years ago

The problem is solved,thank you.