search

elastic / cloud-on-k8s

Elastic Cloud on Kubernetes
Other
2.59k stars 703 forks source link

Can't downsize to the current size after failed volume upsize #4467

Open sebgl opened 3 years ago

sebgl commented 3 years ago

The following situation happened to an Azure user:

  1. User resize their volume claims up
  2. Operator propagates the change to PVC & StatefulSet
  3. AKS can't resize the volume (see https://github.com/Azure/AKS/issues/1477):
error expanding volume "elastic-system/elasticsearch-data-elastic-eck-es-data-0" of plugin "kubernetes.io/azure-disk": compute.DisksClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Cannot resize disk kubernetes-dynamic-pvc-cc190c75-488b-40f7-94ba-9ec8dc3b9e5c while it is attached to running VM /subscriptions/*****************/resourceGroups/******************/providers/Microsoft.Compute/virtualMachineScaleSets/aks-agentpool-93507671-vmss/virtualMachines/aks-agentpool-93507671-vmss_1. Resizing a disk of an Azure Virtual Machine requires the virtual machine to be deallocated. Please stop your VM and retry the operation."
  1. Users try to set back the initial size. This is rejected by the operator:
Failed to apply spec change: handle volume expansion: decreasing storage size is not supported: an attempt was made to decrease storage size for claim elasticsearch-data

At this point it's pretty hard for the user to figure out a way to get back to a clean state, since the StatefulSet has been recreated by ECK with the new size already. Should the operator catch the PVC event error and allow a downsize to go through in that particular case?

leosunmo commented 3 years ago

Just in case someone encounters this like I did, using AKS, Azure managed Kubernetes, here's my work around in rough steps. So far I've only tried to apply new disk size while ECK was running which led to the above error, but I think you can avoid these errors by changing the Elasticsearch CRD desired disk size at step 6 only instead.

  1. Scale down ECK to 0 This is to make sure it doesn't interfere with any of the termination or Elasticsearch settings. kubectl scale statefulset elastic-operator -n elastic-system --replicas 0
  2. Stop Elasticsearch shard allocations and sync This is steps 1,2, and optionally 3 from https://www.elastic.co/guide/en/elasticsearch/reference/current/restart-cluster.html#restart-cluster-full
  3. Delete StatefulSet with orphan cascade option Next you need to remove the ECK-managed StatefulSet for your NodeSet to make sure it doesn't re-create Pods. Important to note here is that we use --cascade=orphan to stop it from removing the Pods. kubectl delete statefulset my-nodeSet --cascade=orphan It's important that you do NOT delete any Services associated with the StatefulSet.
  4. Delete the Pods Next we can remove the Pods. This will leave the PVCs unbound which will cause them to get detached from the Azure VM, which is what the official docs suggest you do, https://github.com/kubernetes-sigs/azuredisk-csi-driver/blob/master/docs/known-issues/sizegrow.md. If you're cautious, delete one Pod first and check that the disk is resized, then move on to the next Pod and so on. I just verified with one Pod/PVC, then deleted all other Pods in my NodeSet.
  5. (Optional) Resize Disks I had a problem where the volume manager was stuck in the errored state and wouldn't resize my disks even after the Pod was deleted and volume detached from the VM, so I edited the PVC again at this point and increased the size by a single GB just to kick-start the resize process, which seemed to work. I had other PVCs eventually resize without requiring a second edit, but it took 10-20 minutes...
  6. Update Elasticsearch CRD Make sure the Elasticsearch CRD is reflecting the new PVC sizes if you had to change them a second time at step 5, or if you didn't resize them before starting the process.
  7. Start ECK again Start ECK again which should trigger it to create a new StatefulSet for the NodeSet, which will reuse the existing PVCs, and boom, you have bigger disks.
  8. Enable shard allocation again After a while (depending on size of cluster) the Cluster will be in a Yellow state again (up from Red when it first starts up). This should be because all primary shards are available, but none of the replica ones, since we disabled shard allocation in step 2. Re-enable it by following step 8 in https://www.elastic.co/guide/en/elasticsearch/reference/current/restart-cluster.html#restart-cluster-full. The cluster should become green after a while, depending on number of indices/shards.

Hopefully this works for others. I am running Elasticsearch 6.8.13 with ECK 1.7.0 in this scenario.

BenjaminHerbert commented 1 year ago

Hello there, I just ran into this issue as well, as I wanted to resize a disk to 4Ti. This failed, as the storage-class I am using was using cachingmode=ReadOnly, which is only supported <4Ti (4095Gi is okay...)

Resetting to the old value or 4095Gi does not work.

Is the above workaround still the way to go, @sebgl?