jprante / elasticsearch-jdbc

JDBC importer for Elasticsearch
Apache License 2.0
2.84k stars 710 forks source link

[SOLVED] Scheduler / Interval configuration problem #687

Closed xavierfacq closed 1 year ago

xavierfacq commented 8 years ago

Hi,

I created a new MySQL import script with the following scheduler configuration : "schedule" : "0 0/10 * ? * *"

I expected that the import runs every 10 minutes (10, 20 ,30, ...) but instead of that, it was running every 30s ... I try to add "interval" : 0, but I have the same problem.

Any help will be appreciated !

Version : elasticsearch-jdbc-1.7.3.0

Metrics logs:
[10:30:30,019][INFO ][metrics.source.plain ][pool-5-thread-1] totalrows = 5725, 30 seconds = 30002 ms, 6086370 = 5,80 MB bytes, 1,04 KB = 1 062 avg size, 190,821 dps, 0,198 MB/s [10:30:30,020][INFO ][metrics.sink.plain ][pool-5-thread-1] 30 seconds = 30005 ms, submitted = 5725, succeeded = 5725, failed = 0, 7846894 = 7,48 MB bytes, 1,34 KB = 1 370 avg size, 190,802 dps, 0,255 MB/s [10:31:00,017][INFO ][metrics.source.plain ][pool-5-thread-1] totalrows = 5725, 1 minute = 60002 ms, 6086370 = 5,80 MB bytes, 1,04 KB = 1 062 avg size, 95,413 dps, 0,099 MB/s [10:31:00,017][INFO ][metrics.sink.plain ][pool-5-thread-1] 1 minute = 60002 ms, submitted = 5725, succeeded = 5725, failed = 0, 7846894 = 7,48 MB bytes, 1,34 KB = 1 370 avg size, 95,413 dps, 0,128 MB/s [10:31:30,017][INFO ][metrics.source.plain ][pool-5-thread-1] totalrows = 5725, 1 minute 30 seconds = 90002 ms, 6086370 = 5,80 MB bytes, 1,04 KB = 1 062 avg size, 63,61 dps, 0,066 MB/s [10:31:30,018][INFO ][metrics.sink.plain ][pool-5-thread-1] 1 minute 30 seconds = 90002 ms, submitted = 5725, succeeded = 5725, failed = 0, 7846894 = 7,48 MB bytes, 1,34 KB = 1 370 avg size, 63,61 dps, 0,085 MB/s [10:32:00,017][INFO ][metrics.source.plain ][pool-5-thread-1] totalrows = 5725, 2 minutes = 120002 ms, 6086370 = 5,80 MB bytes, 1,04 KB = 1 062 avg size, 47,708 dps, 0,05 MB/s [10:32:00,017][INFO ][metrics.sink.plain ][pool-5-thread-1] 2 minutes = 120002 ms, submitted = 5725, succeeded = 5725, failed = 0, 7846894 = 7,48 MB bytes, 1,34 KB = 1 370 avg size, 47,708 dps, 0,064 MB/s

Elatic log : [10:26:14,452][INFO ][importer ][main] schedule with cron expressions [0 0/10 * ? * *]

Thanks !

xavierfacq commented 8 years ago

Ok ! This is not a bug ! This is metric's logs ! Metrics trace every 30s, and after 10 minutes the numbers of submitted documents grows up. So everything is fine.

Sorry for that :-(

jprante commented 8 years ago

Yes, this is a bit confusing. I added the "metrics" logging so the activity can get noticed at all.

xavierfacq commented 8 years ago

It could be interesting to have a metrics configuration to trace "only during/after activity".