search

HbnKing / Flume-ng-Database

Flume-ng sqlsource support data comes from database
9 stars 7 forks source link

flume 请求一次 获取的数据量过大 内存溢出的问题 #1

Open HbnKing opened 5 years ago

HbnKing commented 5 years ago 
2019-02-13 17:06:04,032 (PollableSourceRunner-SQLSource-r1) [INFO - com.hbn.rdb.common.DriverQuery.executeQuery(DriverQuery.java:67)] select * from  newtable  where id > 189606791   ;
Exception in thread "PollableSourceRunner-SQLSource-r1" java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.lang.Integer.valueOf(Integer.java:832)
        at com.mysql.cj.core.conf.ReadableIntegerProperty.getValue(ReadableIntegerProperty.java:41)
        at com.mysql.cj.core.conf.ReadableIntegerProperty.getValue(ReadableIntegerProperty.java:31)
        at com.mysql.cj.mysqla.io.TextRowFactory.createFromPacketPayload(TextRowFactory.java:64)
        at com.mysql.cj.mysqla.io.TextRowFactory.createFromPacketPayload(TextRowFactory.java:40)
        at com.mysql.cj.mysqla.io.ResultsetRowReader.read(ResultsetRowReader.java:83)
        at com.mysql.cj.mysqla.io.ResultsetRowReader.read(ResultsetRowReader.java:38)
        at com.mysql.cj.mysqla.io.MysqlaProtocol.read(MysqlaProtocol.java:1660)
        at com.mysql.cj.mysqla.io.TextResultsetReader.read(TextResultsetReader.java:82)
        at com.mysql.cj.mysqla.io.TextResultsetReader.read(TextResultsetReader.java:43)
        at com.mysql.cj.mysqla.io.MysqlaProtocol.read(MysqlaProtocol.java:1672)
        at com.mysql.cj.mysqla.io.MysqlaProtocol.readAllResults(MysqlaProtocol.java:1718)
        at com.mysql.cj.mysqla.io.MysqlaProtocol.sqlQueryDirect(MysqlaProtocol.java:1006)
        at com.mysql.cj.mysqla.MysqlaSession.sqlQueryDirect(MysqlaSession.java:431)
        at com.mysql.cj.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:1974)
        at com.mysql.cj.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:1936)
        at com.mysql.cj.jdbc.StatementImpl.executeQuery(StatementImpl.java:1422)
        at com.hbn.rdb.common.DriverQuery.executeQuery(DriverQuery.java:70)
        at com.hbn.rdb.source.SQLSourceHelper.executeQuery(SQLSourceHelper.java:245)
        at com.hbn.rdb.source.SQLSource.process(SQLSource.java:79)
        at org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:133)
        at java.lang.Thread.run(Thread.java:748)
HbnKing commented 5 years ago

1建议采用分页模型 2增加flume 内存

HbnKing commented 5 years ago

flume -ng 的 设置 修改 vi bin/flume-ng 修改java ops

################################
# main
################################

# set default params
FLUME_CLASSPATH=""
FLUME_JAVA_LIBRARY_PATH=""
JAVA_OPTS="-Xmx1024m"
LD_LIBRARY_PATH=""

默认 20m