search

apache / druid

Apache Druid: a high performance real-time analytics database.
https://druid.apache.org/
Apache License 2.0
13.4k stars 3.68k forks source link

[0.9.0] Too fast leadership changes in overlord causes multiple leaders and service failure #3046

Closed drcrallen closed 8 years ago

drcrallen commented 8 years ago

It is possible for two overlords in HA mode to get into some sort of leadership fight which causes high cpu load on both instances and general instability of the indexing service.

Logs from overlord 1:

2016-06-01T17:55:33,794 INFO [Curator-LeaderSelector-0] io.druid.indexing.overlord.autoscaling.SimpleResourceManagementStrategy - Started Resource Management Scheduler
2016-06-01T17:55:33,794 INFO [Curator-LeaderSelector-0] io.druid.indexing.overlord.TaskMaster - Bowing out!
2016-06-01T17:55:33,798 ERROR [Curator-LeaderSelector-0] io.druid.indexing.overlord.TaskMaster - Failed to lead: {class=io.druid.indexing.overlord.TaskMaster, exceptionType=class java.lang.reflect.I
nvocationTargetException, exceptionMessage=null}
java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_45]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_45]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_45]
        at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_45]
        at com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:350) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at com.metamx.common.lifecycle.Lifecycle.start(Lifecycle.java:259) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.TaskMaster$1.takeLeadership(TaskMaster.java:134) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector$WrappedListener.takeLeadership(LeaderSelector.java:536) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector.doWork(LeaderSelector.java:399) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector.doWorkLoop(LeaderSelector.java:443) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector.access$100(LeaderSelector.java:64) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:245) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:239) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_45]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_45]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_45]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]
Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@4614612 rejected from java.util.concurrent.ScheduledThreadPoolExecutor@2f19712b[Shutting down, pool size = 1, active threads = 0, queued tasks = 2, completed tasks = 966]
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047) ~[?:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047) ~[?:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) ~[?:1.8.0_45]
        at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326) ~[?:1.8.0_45]
        at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533) ~[?:1.8.0_45]
        at com.metamx.common.concurrent.ScheduledExecutors.scheduleAtFixedRate(ScheduledExecutors.java:159) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at com.metamx.common.concurrent.ScheduledExecutors.scheduleAtFixedRate(ScheduledExecutors.java:135) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at com.metamx.common.concurrent.ScheduledExecutors.scheduleAtFixedRate(ScheduledExecutors.java:121) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.autoscaling.SimpleResourceManagementStrategy.startManagement(SimpleResourceManagementStrategy.java:276) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.autoscaling.SimpleResourceManagementStrategy.startManagement(SimpleResourceManagementStrategy.java:51) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.RemoteTaskRunner.start(RemoteTaskRunner.java:293) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        ... 19 more
2016-06-01T17:55:33,809 ERROR [Curator-LeaderSelector-0] org.apache.curator.framework.recipes.leader.LeaderSelector - The leader threw an exception
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
        at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.TaskMaster$1.takeLeadership(TaskMaster.java:150) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector$WrappedListener.takeLeadership(LeaderSelector.java:536) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector.doWork(LeaderSelector.java:399) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector.doWorkLoop(LeaderSelector.java:443) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector.access$100(LeaderSelector.java:64) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:245) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:239) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_45]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_45]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_45]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_45]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_45]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_45]
        at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_45]
        at com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:350) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at com.metamx.common.lifecycle.Lifecycle.start(Lifecycle.java:259) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.TaskMaster$1.takeLeadership(TaskMaster.java:134) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        ... 12 more
Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@4614612 rejected from java.util.concurrent.ScheduledThreadPoolExecutor@2f19712b[Shutting down, pool size = 1, active threads = 0, queued tasks = 2, completed tasks = 966]
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047) ~[?:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) ~[?:1.8.0_45]
        at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326) ~[?:1.8.0_45]
        at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533) ~[?:1.8.0_45]
        at com.metamx.common.concurrent.ScheduledExecutors.scheduleAtFixedRate(ScheduledExecutors.java:159) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at com.metamx.common.concurrent.ScheduledExecutors.scheduleAtFixedRate(ScheduledExecutors.java:135) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at com.metamx.common.concurrent.ScheduledExecutors.scheduleAtFixedRate(ScheduledExecutors.java:121) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.autoscaling.SimpleResourceManagementStrategy.startManagement(SimpleResourceManagementStrategy.java:276) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.autoscaling.SimpleResourceManagementStrategy.startManagement(SimpleResourceManagementStrategy.java:51) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.RemoteTaskRunner.start(RemoteTaskRunner.java:293) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_45]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_45]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_45]
        at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_45]
        at com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:350) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at com.metamx.common.lifecycle.Lifecycle.start(Lifecycle.java:259) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.TaskMaster$1.takeLeadership(TaskMaster.java:134) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        ... 12 more
2016-06-01T17:55:44,008 INFO [Curator-LeaderSelector-0] io.druid.indexing.overlord.TaskMaster - By the power of Grayskull, I have the power!

Logs from overlord 2:

2016-06-01T17:55:25,231 INFO [Curator-LeaderSelector-0] io.druid.indexing.overlord.autoscaling.SimpleResourceManagementStrategy - Started Resource Management Scheduler
2016-06-01T17:55:25,231 INFO [Curator-LeaderSelector-0] io.druid.indexing.overlord.TaskMaster - Bowing out!
2016-06-01T17:55:25,231 ERROR [Curator-LeaderSelector-0] io.druid.indexing.overlord.TaskMaster - Failed to lead: {class=io.druid.indexing.overlord.TaskMaster, exceptionType=class java.lang.reflect.InvocationTargetException, exceptionMessage=null}
java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_45]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_45]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_45]
        at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_45]
        at com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:350) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at com.metamx.common.lifecycle.Lifecycle.start(Lifecycle.java:259) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.TaskMaster$1.takeLeadership(TaskMaster.java:134) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector$WrappedListener.takeLeadership(LeaderSelector.java:536) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector.doWork(LeaderSelector.java:399) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector.doWorkLoop(LeaderSelector.java:443) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector.access$100(LeaderSelector.java:64) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:245) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:239) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_45]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_45]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_45]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]
Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@1ade306d rejected from java.util.concurrent.ScheduledThreadPoolExecutor@20d11e35[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 17550]
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047) ~[?:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) ~[?:1.8.0_45]
        at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326) ~[?:1.8.0_45]
        at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533) ~[?:1.8.0_45]
        at com.metamx.common.concurrent.ScheduledExecutors.scheduleAtFixedRate(ScheduledExecutors.java:159) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at com.metamx.common.concurrent.ScheduledExecutors.scheduleAtFixedRate(ScheduledExecutors.java:135) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at com.metamx.common.concurrent.ScheduledExecutors.scheduleAtFixedRate(ScheduledExecutors.java:121) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.autoscaling.SimpleResourceManagementStrategy.startManagement(SimpleResourceManagementStrategy.java:276) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.autoscaling.SimpleResourceManagementStrategy.startManagement(SimpleResourceManagementStrategy.java:51) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.RemoteTaskRunner.start(RemoteTaskRunner.java:293) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        ... 19 more
2016-06-01T17:55:25,232 ERROR [Curator-LeaderSelector-0] org.apache.curator.framework.recipes.leader.LeaderSelector - The leader threw an exception
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
        at com.google.common.base.Throwables.propagate(Throwables.java:160) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.TaskMaster$1.takeLeadership(TaskMaster.java:150) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector$WrappedListener.takeLeadership(LeaderSelector.java:536) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector.doWork(LeaderSelector.java:399) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector.doWorkLoop(LeaderSelector.java:443) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector.access$100(LeaderSelector.java:64) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:245) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:239) [druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_45]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_45]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_45]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_45]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_45]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_45]
        at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_45]
        at com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:350) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at com.metamx.common.lifecycle.Lifecycle.start(Lifecycle.java:259) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.TaskMaster$1.takeLeadership(TaskMaster.java:134) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        ... 12 more
Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@1ade306d rejected from java.util.concurrent.ScheduledThreadPoolExecutor@20d11e35[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 17550]
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047) ~[?:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823) ~[?:1.8.0_45]
        at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326) ~[?:1.8.0_45]
        at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533) ~[?:1.8.0_45]
        at com.metamx.common.concurrent.ScheduledExecutors.scheduleAtFixedRate(ScheduledExecutors.java:159) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at com.metamx.common.concurrent.ScheduledExecutors.scheduleAtFixedRate(ScheduledExecutors.java:135) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at com.metamx.common.concurrent.ScheduledExecutors.scheduleAtFixedRate(ScheduledExecutors.java:121) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.autoscaling.SimpleResourceManagementStrategy.startManagement(SimpleResourceManagementStrategy.java:276) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.autoscaling.SimpleResourceManagementStrategy.startManagement(SimpleResourceManagementStrategy.java:51) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.RemoteTaskRunner.start(RemoteTaskRunner.java:293) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_45]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_45]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_45]
        at java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_45]
        at com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:350) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at com.metamx.common.lifecycle.Lifecycle.start(Lifecycle.java:259) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        at io.druid.indexing.overlord.TaskMaster$1.takeLeadership(TaskMaster.java:134) ~[druid-selfcontained-0.9.0-rc3-mmx3.jar:0.9.0-rc3-mmx3]
        ... 12 more
2016-06-01T17:55:33,810 INFO [Curator-LeaderSelector-0] io.druid.indexing.overlord.TaskMaster - By the power of Grayskull, I have the power!
gianm commented 8 years ago

@drcrallen does the [0.9.0] tag mean this is a regression in 0.9.0 or just that this is the version you noticed it in?

drcrallen commented 8 years ago

It is the version I noticed it in.

drcrallen commented 8 years ago

Didn't want people to think it was a 0.9.1 thing :)