Persistent data management

[mabilinab]

Hi there, I have found very little info on how the operator manages data. I have used volumeClaimTemplates to dynamically provision storage, however, I'm not quite sure of how the operator manages pvcs:

• It seems like when you change the volumeClaimTemplate storage, it is not taken into account and the pods keep using the same PVC which is not updated
• When scaling down the number of nodes, does the operator take care of redistributing data from the exiting data node to other nodes?
• When you decrease the number of replicas, the operator does not delete the PVC, which creates additional cost for unused provisioned volumes that stay unattached (hetzner volumes). Is this normal?

Does anyone have any tips or info on this? Thank you!

[otrosien]

Hello @mabilinab,

the thing is, ES-Operator doesn't actually manage any persistent volumes, it all gets delegated to the underlying StatefulSet. You should check the StatefulSet properties so that the volumeClaim gets removed (I'm not too deep into this).

Resizing is something not every Kubernetes installation supports, but you can find discussions with how-tos, for example here: https://serverfault.com/questions/955293/how-to-increase-disk-size-in-a-stateful-set

ES-Operator defaults to draining the node before termination, which is different from what the ECK does. If I'm not mistaken, ECK assumes persistent volumes and considers a temporary reduced availability of data being fine while the data node is supposed to come back online. When draining, the redistribution of data is delegated to Elasticsearch itself. It tries to keep a balanced shard-per-node ratio, based on a heuristic that can be tweaked, but they don't recommend you to do so: https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html#shards-rebalancing-heuristics

There are other constraints that will cause Elasticsearch not to colocate more shards onto the same node. Amongst them:

• Another copy of the same shard is already on this node
• If it would cause an imbalance for the cluster-level shard distribution (https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html#shard-allocation-awareness) or violate index-level or node-level shard allocation settings (https://www.elastic.co/guide/en/elasticsearch/reference/current/allocation-total-shards.html), for which you can use "cluster-allocation explain" to get an understanding why: https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-allocation-explain.html
• Other constraints like high disk watermarks, or mismatch on required node labels etc.

Hope this helps.