Allow possibility to update a storage size of a Node

[vDMG]

Today if I try to update the storage size of my volumeClaimTemplates on a NodeSpec and try to apply my change the operator return the following error :

{"level":"error","ts":1585819955.9165432,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"druid-controller","request":"partner-stats/druid-partner-stats","error":"Failed to update [StatefulSet:druid-druid-partner-stats-historicals] due to [StatefulSet.apps \"druid-druid-partner-stats-historicals\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden].","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/druid-operator/vendor/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/druid-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/druid-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/druid-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/druid-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/druid-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/druid-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}

Most of Kubernetes cluster now handle PVC extension and need a pod restart. We could manage this feature.

[AdheipSingh]

Do you mind sending your storage class spec @vDMG

[vDMG]

I mean if I previously applied this conf for my historicalNodeSpec :

volumeClaimTemplates:
        - metadata:
            name: data-volume
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 250Gi
            storageClassName: ssd

and try to update it with

volumeClaimTemplates:
        - metadata:
            name: data-volume
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 300Gi
            storageClassName: ssd

I'm getting the error below.

my storage class spec

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ssd
parameters:
  type: pd-ssd
provisioner: kubernetes.io/gce-pd
reclaimPolicy: Delete
volumeBindingMode: Immediate

[vDMG]

kind of related to this issue https://github.com/kubernetes/kubernetes/issues/68737

[AdheipSingh]

try adding this parameter to your storage class @vDMG

allowVolumeExpansion: true

[himanshug]

operator supports the update as long as StatefulSet itself support same. For such cases, when I know what I am doing, workaround I use, I delete the corresponding StatefulSet after updating Druid spec, operator then recreates it.

[AdheipSingh]

https://kubernetes.io/blog/2018/07/12/resizing-persistent-volumes-using-kubernetes/ Might be useful, all depends on your sc and cloud provider

[Rep1AI]

I delete the corresponding StatefulSet after updating Druid spec, the operator then recreates it.

That's against operators' goals... The operator is to eliminate or reduce the operation by us and automate the most things from scaling and other operations right?

[himanshug]

@RohitR1 you are right and this is not ideal, things should be improved wherever, case by case, a good alternative is possible.

allowVolumeExpansion: true sounds like the right thing to do.

[neeleshkorade]

Unfortunately, my storage class spec already has allowVolumeExpansion: true. However, the PVC still won't resize when I update the size in the druid cluster spec.

@AdheipSingh @himanshug has this worked for you/anyone here?

[AdheipSingh]

@neeleshkorade

To enable this feature scalePvcSts needs to be enabled to true.

https://github.com/druid-io/druid-operator/blob/master/docs/features.md#volume-expansion-of-druid-nodes-running-as-statefulsets

https://github.com/druid-io/druid-operator/blob/master/apis/druid/v1alpha1/druid_types.go#L43