Open astelmashenko opened 1 month ago
I'm also studying house keeping mechanism now I think it's a good idea to have elasticsearch ILM set on index patterns, though it's not implemented yet in conductor so you have to manually configure it on elasticsearch
Describe the bug After certain amount of time shards problem raised. We checked ES and found out around 2300 shards, names like conductor_task_log_20221004.
Investigating code I see
Where updateIndexesNames creates new index every week. Can someone explain why does it change names for indices?
I'd like to change this behavior, because we reaching limits on shards.
I also checked below methods are not used anywhere:
so it is probably safe to stop indexing them:
However to stop creating new indices we need to change this config:
And provision indices for conductor manually.
Details Conductor version: 3.18.0+ Persistence implementation: Postgres Queue implementation: Postgres Lock: Redis
To Reproduce Steps to reproduce the behavior: Just use elastic for a while
Expected behavior Data is cleaned up periodically.