search

Netflix / conductor-community

Apache License 2.0
61 stars 72 forks source link

zookeeper lock znode not getting cleared after workflows are finalized #246

Open mithlesh135 opened 1 year ago

mithlesh135 commented 1 year ago

Describe the bug As zookeeper takes lock for any operation, it creates znodes in below format /conductor/conductor-lock/workflow-uuid/sequential-epephermal-uuid

Now these epephermal nodes get deleted but the the znode at workflow uuid is persistence once and is not getting deleted, and and we have more and more workflows getting processed, number of these persistence znodes increase inside the parent conductor-lock znode. Zookeeper suggests max size of znode to be 1 MB, this ever growing znode is a problem.

To Reproduce Steps to reproduce the behavior:

  1. Go to conductor
  2. connect to zookeeper server
  3. run a workflow and check znodes getting created for workflow
  4. even after worklow execution has completed, workflow znode stays

Expected behavior workflow znode should be deleted after workflow is finalized

v1r3n commented 1 year ago

Hi @mithlesh135 I would recommend using Redis for locks. Zookeeper support exists for the legacy reasons and is quite a heavy solution to maintain. Redis based locks are going to be much more lightweight.