Open astelmashenko opened 1 year ago
Investigating code I see
private void createIndexesTemplates() {
try {
initIndexesTemplates();
updateIndexesNames();
Executors.newScheduledThreadPool(1).scheduleAtFixedRate(this::updateIndexesNames, 0, 1, TimeUnit.HOURS);
} catch (Exception e) {
logger.error("Error creating index templates!", e);
}
}
See this line
Executors.newScheduledThreadPool(1).scheduleAtFixedRate(this::updateIndexesNames, 0, 1, TimeUnit.HOURS);
Where updateIndexesNames
creates new index every week.
Can someone explain why does it change names for indices?
I'd like to change this behavior, because we reaching limits on shards.
I also checked below methods are not used anywhere:
IndexDAO.getEventExecutions
IndexDAO.getMessages
so it is probably safe to stop indexing them:
conductor.app.eventMessageIndexingEnabled=false
conductor.app.eventExecutionIndexingEnabled=false
However to stop creating new indices we need to change this config:
conductor.elasticsearch.autoIndexManagementEnabled=false
And provision indices for conductor manually.
Can anybody comment on this?
Hi @astelmashenko , Thanks for reporting this. We will add support for archival from ES also. Let me know if this works, or we can chat here for more realtime collaboration.
@manan164 ,
We will add support for archival from ES
Yes, it will help.
Additionally, I'd add reaction to indexing flag, e.g. if conductor.app.eventMessageIndexingEnabled=false
is set then do not create new indices once a week. Maybe think also about changing period of creating new indices, e.g. WEEK, MONTH, YEAR
Thanks
@manan164 , are there any work done on archival of indicies?
Describe the bug After certain amount of time shards problem raised. We checked ES and found out around 2300 shards, we names like
conductor_task_log_20221004
.Details Conductor version: 3.12.2 Persistence implementation: Postgres Queue implementation: Postgres Lock: Redis
To Reproduce Steps to reproduce the behavior:
Expected behavior Data is cleaned up periodically.