search

nirmata / workflow

A ZooKeeper and Curator based distributed workflow management library that enables distributed task workflows.
http://nirmata.github.io/workflow
Apache License 2.0
96 stars 48 forks source link

SchedulerSelector throws IllegalStateException on exiting the runLoop #52

Open beloyar777 opened 1 year ago

beloyar777 commented 1 year ago

I'm very new to ZK/Curator and was just executing your example. It seems that your SchedulerSelector tries to take leadership after curator is closed. May be a proper closing of SchedulerSelector should be added to the usage example or are these 39 errors intended?

Getting the following 39 IllegalStateExceptions while executing your example:

2023-09-12 09:40:57,206 INFO  com.nirmata.workflow.queue.zookeeper.SimpleQueue [Curator-SimpleQueue-0] - Exiting runLoop
2023-09-12 09:40:57,206 INFO  com.nirmata.workflow.queue.zookeeper.SimpleQueue [Curator-SimpleQueue-0] - Exiting runLoop
2023-09-12 09:40:57,206 INFO  com.nirmata.workflow.queue.zookeeper.SimpleQueue [Curator-SimpleQueue-0] - Exiting runLoop
2023-09-12 09:40:57,206 INFO  com.nirmata.workflow.queue.zookeeper.SimpleQueue [Curator-SimpleQueue-0] - Exiting runLoop
2023-09-12 09:40:57,206 INFO  com.nirmata.workflow.queue.zookeeper.SimpleQueue [Curator-SimpleQueue-0] - Exiting runLoop
2023-09-12 09:40:57,206 INFO  com.nirmata.workflow.queue.zookeeper.SimpleQueue [Curator-SimpleQueue-0] - Exiting runLoop
2023-09-12 09:40:57,207 INFO  com.nirmata.workflow.queue.zookeeper.SimpleQueue [Curator-SimpleQueue-0] - Exiting runLoop
2023-09-12 09:40:57,207 INFO  com.nirmata.workflow.queue.zookeeper.SimpleQueue [Curator-SimpleQueue-0] - Exiting runLoop
2023-09-12 09:40:57,207 INFO  com.nirmata.workflow.queue.zookeeper.SimpleQueue [Curator-SimpleQueue-0] - Exiting runLoop
2023-09-12 09:40:57,206 INFO  com.nirmata.workflow.queue.zookeeper.SimpleQueue [Curator-SimpleQueue-0] - Exiting runLoop
2023-09-12 09:40:57,207 INFO  org.apache.curator.framework.imps.CuratorFrameworkImpl [Curator-Framework-0] - backgroundOperationsLoop exiting

// the following exception is logged 38 times
2023-09-12 09:40:57,208 ERROR org.apache.curator.framework.imps.CuratorFrameworkImpl [Curator-LeaderSelector-0] - Background exception was not retry-able or retry gave up
java.lang.IllegalStateException: Client is not started
                at org.apache.curator.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:502) ~[curator-client-5.5.0.jar:5.5.0]
                at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:139) ~[curator-client-5.5.0.jar:5.5.0]
                at org.apache.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:649) ~[curator-framework-5.5.0.jar:5.5.0]
                at org.apache.curator.framework.imps.NamespaceFacade.getZooKeeper(NamespaceFacade.java:115) ~[curator-framework-5.5.0.jar:5.5.0]
                at org.apache.curator.framework.imps.RemoveWatchesBuilderImpl.performBackgroundOperation(RemoveWatchesBuilderImpl.java:337) ~[curator-framework-5.5.0.jar:5.5.0]
                at org.apache.curator.framework.imps.OperationAndData.callPerformBackgroundOperation(OperationAndData.java:84) ~[curator-framework-5.5.0.jar:5.5.0]
                at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:1008) ~[curator-framework-5.5.0.jar:5.5.0]
                at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:667) ~[curator-framework-5.5.0.jar:5.5.0]
                at org.apache.curator.framework.imps.NamespaceFacade.processBackgroundOperation(NamespaceFacade.java:121) ~[curator-framework-5.5.0.jar:5.5.0]
                at org.apache.curator.framework.imps.RemoveWatchesBuilderImpl.pathInBackground(RemoveWatchesBuilderImpl.java:229) ~[curator-framework-5.5.0.jar:5.5.0]
                at org.apache.curator.framework.imps.RemoveWatchesBuilderImpl.internalRemoval(RemoveWatchesBuilderImpl.java:84) ~[curator-framework-5.5.0.jar:5.5.0]
                at org.apache.curator.framework.imps.WatcherRemovalManager.removeWatchers(WatcherRemovalManager.java:67) ~[curator-framework-5.5.0.jar:5.5.0]
                at org.apache.curator.framework.imps.WatcherRemovalFacade.removeWatchers(WatcherRemovalFacade.java:68) ~[curator-framework-5.5.0.jar:5.5.0]
                at org.apache.curator.framework.recipes.cache.PathChildrenCache.close(PathChildrenCache.java:384) ~[curator-recipes-5.5.0.jar:5.5.0]
                at org.apache.curator.shaded.com.google.common.io.Closeables.close(Closeables.java:79) ~[curator-client-5.5.0.jar:5.5.0]
                at org.apache.curator.utils.CloseableUtils.closeQuietly(CloseableUtils.java:59) ~[curator-client-5.5.0.jar:5.5.0]
                at com.nirmata.workflow.details.Scheduler.run(Scheduler.java:173) ~[nirmata-workflow-0.9.7.jar:?]
                at com.nirmata.workflow.details.SchedulerSelector.takeLeadership(SchedulerSelector.java:97) ~[nirmata-workflow-0.9.7.jar:?]
                at com.nirmata.workflow.details.SchedulerSelector.access$000(SchedulerSelector.java:33) ~[nirmata-workflow-0.9.7.jar:?]
                at com.nirmata.workflow.details.SchedulerSelector$1.takeLeadership(SchedulerSelector.java:55) ~[nirmata-workflow-0.9.7.jar:?]
                at org.apache.curator.framework.recipes.leader.LeaderSelector$WrappedListener.takeLeadership(LeaderSelector.java:594) ~[curator-recipes-5.5.0.jar:5.5.0]
                at org.apache.curator.framework.recipes.leader.LeaderSelector.doWork(LeaderSelector.java:452) ~[curator-recipes-5.5.0.jar:5.5.0]
                at org.apache.curator.framework.recipes.leader.LeaderSelector.doWorkLoop(LeaderSelector.java:506) ~[curator-recipes-5.5.0.jar:5.5.0]
                at org.apache.curator.framework.recipes.leader.LeaderSelector.access$200(LeaderSelector.java:66) ~[curator-recipes-5.5.0.jar:5.5.0]
                at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:251) ~[curator-recipes-5.5.0.jar:5.5.0]
                at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:244) ~[curator-recipes-5.5.0.jar:5.5.0]
                at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
                at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
                at java.lang.Thread.run(Thread.java:833) ~[?:?]
…
2023-09-12 09:40:57,223 INFO  com.nirmata.workflow.details.SchedulerSelector [Curator-LeaderSelector-0] - cab-wsv-demaawo is no longer the scheduler
2023-09-12 09:40:57,224 ERROR org.apache.curator.framework.recipes.leader.LeaderSelector [Curator-LeaderSelector-0] - The leader threw an exception
java.lang.IllegalStateException: Expected state [STARTED] was [STOPPED]
                at org.apache.curator.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:821) ~[curator-client-5.5.0.jar:5.5.0]
                at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkState(CuratorFrameworkImpl.java:457) ~[curator-framework-5.5.0.jar:5.5.0]
                at org.apache.curator.framework.imps.CuratorFrameworkImpl.delete(CuratorFrameworkImpl.java:477) ~[curator-framework-5.5.0.jar:5.5.0]
                at org.apache.curator.framework.recipes.locks.LockInternals.deleteOurPath(LockInternals.java:348) ~[curator-recipes-5.5.0.jar:5.5.0]
                at org.apache.curator.framework.recipes.locks.LockInternals.releaseLock(LockInternals.java:125) ~[curator-recipes-5.5.0.jar:5.5.0]
                at org.apache.curator.framework.recipes.locks.InterProcessMutex.release(InterProcessMutex.java:154) ~[curator-recipes-5.5.0.jar:5.5.0]
                at org.apache.curator.framework.recipes.leader.LeaderSelector.doWork(LeaderSelector.java:477) ~[curator-recipes-5.5.0.jar:5.5.0]
                at org.apache.curator.framework.recipes.leader.LeaderSelector.doWorkLoop(LeaderSelector.java:506) ~[curator-recipes-5.5.0.jar:5.5.0]
                at org.apache.curator.framework.recipes.leader.LeaderSelector.access$200(LeaderSelector.java:66) ~[curator-recipes-5.5.0.jar:5.5.0]
                at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:251) ~[curator-recipes-5.5.0.jar:5.5.0]
                at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:244) ~[curator-recipes-5.5.0.jar:5.5.0]
                at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
                at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
                at java.lang.Thread.run(Thread.java:833) ~[?:?]
2023-09-12 09:40:57,313 INFO  org.apache.zookeeper.ZooKeeper [main] - Session: 0x1008ba19a880016 closed
2023-09-12 09:40:57,313 INFO  org.apache.zookeeper.ClientCnxn [main-EventThread] - EventThread shut down for session: 0x1008ba19a880016

Process finished with exit code 0

ZK server 3.6.4 ZK clients tested 3.6.4, 3.7.1, 3.8.2, 3.9.0 curator-recipes 5.2.0, 5.5.0

Randgalt commented 1 year ago

Hi - thanks for the note. tbh - I haven't looked at this library in a very long time. I'll try to investigate when I get a chance.