pebrc
commented
2 years ago The default value for allocation_delay is still 5 minutes. I realise that the documentation I linked to in the comment does not (no longer?) mention that but the code is here:
I do however understand that there might be situations where a fixed 5 minute value is not enough. When you say you would like this to be configurable on the operator level then am I right in assuming you would like to have per cluster overrides for this? We could support an annotation per cluster or even an explicit field in the CRD to set this per cluster
tommasopozzetti
Bug Report
After upgrading the operator to version 2+, we started noticing rolling upgrade procedures being much slower. Upon looking more into it, the utilization of the shutdown API instead of the legacy transient settings was causing Elasticsearch to start reallocation of indices before the restarted nodes were rejoining the cluster, thus taking a very long time to heal after each node restart.
This seems to be caused by the fact that the operator now relies on a timeout rather than waiting for the pod to be back up and manually restart allocations and therefore, if that timeout expires before the node rejoins the cluster, the shards start getting reallocated.
I believe all this is the expected behavior of the shutdown API, however, I think there might be a misalignment in terms of the default timeout. The shutdown API allows to specify a custom timeout (allocation_delay) when performing the shutdown call, but that doesn't seem to get used by the operator, which instead relies on the default allocation_delay of the indices as seen by the comment here. The comment seems to suggest the operator should rely on the default timeout of 5m, however, Elasticsearch documentation reports the default timeout to be 1m (see here.
1 minute definitely seems to be a very short timeout in Kubernetes where if anything in the lifecycle of the pod (termination, unmounting of volumes, scheduling of the new pod, image pulling, mounting of volumes, initcontainers....) takes a little too long to execute, there is a high risk of the node not rejoining the cluster in time (which is what has happened to us for multiple nodes).
To further confirm this is the issue, I have changed every index in the cluster to have a higher delayed_timeout (10m) and ran another upgrade and this time everything completed successfully without any delays. This solution, however, is very tedious because delayed_timeout is a per index setting (so it needs to be applied to every index and needs to be configured on every index template to ensure new indices inherit it). Furthermore, this setting is meant more for unplanned issues where possibly the 1m default response time makes sense, while it might make sense to have a higher one for planned maintenances if it is known that the maintenance could take longer.
My proposal is to set a custom allocation_delay in the shutdown call to override the default one and set a sane default of at least 5m in that call. Furthermore, possibly make that value configurable on the operator level so that, for clusters with particular needs that might expect an even higher timeout to be needed, it is possible to override it and set a proper timeout that matches the needs of that cluster.