search

keedio / flume-ng-sql-source

Flume Source to import data from SQL Databases
Apache License 2.0
264 stars 164 forks source link

Flume randomly stopped fetching data from source #27

Closed martinuselvin closed 6 years ago

martinuselvin commented 7 years ago

Hello,

There is a bug(?) that really bothering me when i use this plugin. To be precise, the thread responsible to read the data from source is completely stopped (Flume itself still running). The thread name is PollableSourceRunner-SQLSource-[source_name]. I have set the flume log level to DEBUG, but there is no error at all. I am using memoryChannel and kafka sink for its configuration.

I hope you can reproduce this bug, as it is really randomly appear. For me, it can take some hours, and few days for the other try.

Regards,

elvin

mvalleavila commented 7 years ago

Hello Elvin,

I will review the source to try to reproduce the bug and solve it. Thanks for get it.

Regards, Marcelo

...................................................................

Marcelo Valle Ávila

mvalle@keedio.com |+34 630 371 156

www.keedio.com

...................................................................

2016-09-27 7:11 GMT+02:00 martinuselvin notifications@github.com:

Hello,

There is a bug(?) that really bothering me when i use this plugin. To be precise, the thread responsible to read the data from source is completely stopped (Flume itself still running). The thread name is PollableSourceRunner-SQLSource-[source_name]. I have set the flume log level to DEBUG, but there is no error at all. I am using memoryChannel and kafka sink for its configuration.

I hope you can reproduce this bug, as it is really randomly appear. For me, it can take some hours, and few days for the other try.

Regards,

elvin

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/keedio/flume-ng-sql-source/issues/27, or mute the thread https://github.com/notifications/unsubscribe-auth/AERdiPqxVYPzwh44w-J0B9CkAFSnRIx-ks5quKVkgaJpZM4KHQyz .

martinuselvin commented 7 years ago

Thanks Marcelo. Looking forward for your answer! For additional information, i am using flume version 1.5.2.2.4.2.0-258 and flume-ng-sql-source version 1.4.2.

martinuselvin commented 7 years ago

Hi Marcelo,

Did you able to reproduce the bug?

Regards, Elvin

lucarosellini commented 7 years ago

@martinuselvin we're trying to reproduce the bug, could you please provide us the configuration you're using (without any real endpoint, usernames and passwords, of course)? Furthermore, do you have an idea of how many events per second your agent is handling? This would be really helpful to us.

martinuselvin commented 7 years ago

flume.txt Here is the config file. As for how many events per second, it can handle as much as the batch size i state in config file (1000) (I hope this is the right answer).

Regards, Elvin

martinuselvin commented 7 years ago

Sorry, I clicked the wrong button, didn't mean to close it.

rishit2016 commented 7 years ago

Hi martin,

Even i am getting below error Exception in thread "PollableSourceRunner-SQLSource-sql-source" java.lang.AbstractMethodError: org.keedio.flume.source.SQLSource.getMaxBackOffSleepInterval()J at org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:154) at java.lang.Thread.run(Thread.java:745)

Note that this is happening only at around midnight time. rest whole day it is running fine.

please advise.

thanks, Rishit Shah

lufeng1102 commented 7 years ago

Hi rfshit

Do you fix this issue. I have also meet this problem using latest version of flume-ng-sql-source and flume version is 1.7.0.

exception log like this : Exception in thread "PollableSourceRunner-SQLSource-sqlSource_product" java.lang.AbstractMethodError: org.keedio.flume.source.SQLSource.getMaxBackOffSleepInterval()J at org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:150) at java.lang.Thread.run(Thread.java:748)

lazaromedina commented 6 years ago

Same problem as #34 Since Apache-Flume 1.7.0, PollableSource implements new methods for managing backoff times. Release flume-sql-source 1.5.0 is compatible with upper versions Apache flume since 1.7.0. https://issues.apache.org/jira/browse/FLUME-2729 Thanks

skt0001 commented 6 years ago

Hi, I am getting an error while reading logs from rabbitmq to HDFS path using Flume.

Please can somebody help me to rectify the issue.

Exception in thread "PollableSourceRunner-RabbitMQSource-rabbitmq-source1" java.lang.AbstractMethodError: org.apache.flume.source.rabbitmq.RabbitMQSource.getBackOffSleepIncrement()J at org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:142) at java.lang.Thread.run(Thread.java:748)

veve90 commented 6 years ago

I have the same problem, Flume stops randomly, but I have nothing in my logs..

skt0001 commented 6 years ago

Hi, Still I have no clue to fix the below mentioned error. Please can somebody help me to rectify the issue. The Flume agent runs and pulls the file in the first 1-2 hours, and later prevents it from dragging logs from rabbitmq into the hdfs path. After running for 1-2 hours, it throws an error in which no file is being created. So every time, we need to stop the agent and start it again.

Exception in thread "PollableSourceRunner-RabbitMQSource-rabbitmq-source1" java.lang.AbstractMethodError: org.apache.flume.source.rabbitmq.RabbitMQSource.getBackOffSleepIncrement()J at org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:142) at java.lang.Thread.run(Thread.java:748)

Thanks,