Cant' upgrade loki-stack due to "failed to replace pvc"

[stephen-lazarionok]

Can't upgrade the helm release. Getting the following error:

helm upgrade --install loki-stack grafana/loki-stack -f kube-prom-stack-values.yaml

Error: UPGRADE FAILED: cannot patch "loki-stack" with kind StatefulSet: StatefulSet.apps "loki-stack" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden

When trying to force it, then having the following issue:

helm upgrade --install loki-stack grafana/loki-stack -f kube-prom-stack-values.yaml --force

Error: UPGRADE FAILED: failed to replace object: PersistentVolumeClaim "loki-stack-grafana" is invalid: spec: Forbidden: is immutable after creation except resources.requests for bound claims && failed to replace object: PersistentVolumeClaim "loki-stack-prometheus-alertmanager" is invalid: spec: Forbidden: is immutable after creation except resources.requests for bound claims && failed to replace object: PersistentVolumeClaim "loki-stack-prometheus-server" is invalid: spec: Forbidden: is immutable after creation except resources.requests for bound claims && failed to replace object: Service "loki-stack-grafana" is invalid: spec.clusterIP: Invalid value: "": field is immutable && failed to replace object: Service "loki-stack" is invalid: spec.clusterIP: Invalid value: "": field is immutable && failed to replace object: Service "loki-stack-kube-state-metrics" is invalid: spec.clusterIP: Invalid value: "": field is immutable && failed to replace object: Service "loki-stack-prometheus-alertmanager" is invalid: spec.clusterIP: Invalid value: "": field is immutable && failed to replace object: Service "loki-stack-prometheus-pushgateway" is invalid: spec.clusterIP: Invalid value: "": field is immutable && failed to replace object: Service "loki-stack-prometheus-server" is invalid: spec.clusterIP: Invalid value: "": field is immutable && failed to replace object: StatefulSet.apps "loki-stack" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden

cat kube-prom-stack-values.yaml
grafana:
  enabled: true
  persistence:
    type: pvc
    enabled: true
    # storageClassName: default
    accessModes:
      - ReadWriteOnce
    size: 5Gi
    # annotations: {}
    finalizers:
      - kubernetes.io/pvc-protection
    # selectorLabels: {}
    # subPath: ""
    # existingClaim:

prometheus:
  enabled: true
  config:
    table_manager:
      retention_deletes_enabled: true
      retention_period: 720h

loki:
  persistence:
    enabled: true
    accessModes:
      - ReadWriteOnce
    size: 30Gi
    #annotations: { }
  config:
    chunk_store_config:
      max_look_back_period: 840h
    table_manager:
      retention_deletes_enabled: true
      retention_period: 840h # should be divided by 168

[Escaflow]

Run in the same issue, try a --dry-run and check the volumes:

[...]
      volumes:
        - name: config
          secret:
            secretName: loki
        - name: storage
          emptyDir: {}

oddly enough if you overwrite it on the cli it works for me, try this:

helm upgrade --install loki-stack grafana/loki-stack -f kube-prom-stack-values.yaml \
--set loki.persistence.enabled=true \
--set loki.persistence.size=30Gi

[villesau]

I have the same problem with grafana installation so not a loki-stack issue only.

E: In helm repo someone proposes following fix: https://github.com/helm/charts/issues/20389#issuecomment-583854511