thanos-store without volumeClaimTemplate not working

[itspngu]

According to the docs, backing thanos-store with a PVC is optional:

It [thanos-store] acts primarily as an API gateway and therefore does not need significant amounts of local disk space. It joins a Thanos cluster on startup and advertises the data it can access. It keeps a small amount of information about all remote blocks on local disk and keeps it in sync with the bucket. This data is generally safe to delete across restarts at the cost of increased startup times.

When compiling a jsonnet configuration not containing config.volumeClaimTemplate, the statefulset is never created because the "data" volume can not be found:

$ kubectl -n monitoring describe statefulset/thanos-store
Name:               thanos-store
Namespace:          monitoring
CreationTimestamp:  Mon, 22 Nov 2021 15:10:46 +0000
Selector:           app.kubernetes.io/component=object-store-gateway,app.kubernetes.io/instance=thanos-store,app.kubernetes.io/name=thanos-store
Labels:             app.kubernetes.io/component=object-store-gateway
                    app.kubernetes.io/instance=thanos-store
                    app.kubernetes.io/name=thanos-store
                    app.kubernetes.io/version=v0.22.0
                    kustomize.toolkit.fluxcd.io/name=kube-prometheus-thanos
                    kustomize.toolkit.fluxcd.io/namespace=monitoring
Annotations:        <none>
Replicas:           1 desired | 0 total
Update Strategy:    RollingUpdate
  Partition:        0
Pods Status:        0 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:           app.kubernetes.io/component=object-store-gateway
                    app.kubernetes.io/instance=thanos-store
                    app.kubernetes.io/name=thanos-store
                    app.kubernetes.io/version=v0.22.0
  Service Account:  thanos-store
  Containers:
   thanos-store:
    Image:       quay.io/thanos/thanos:v0.22.0
    Ports:       10901/TCP, 10902/TCP
    Host Ports:  0/TCP, 0/TCP
    Args:
      store
      --log.level=info
      --log.format=logfmt
      --data-dir=/var/thanos/store
      --grpc-address=0.0.0.0:10901
      --http-address=0.0.0.0:10902
      --objstore.config=$(OBJSTORE_CONFIG)
      --ignore-deletion-marks-delay=24h
    Liveness:   http-get http://:10902/-/healthy delay=0s timeout=1s period=30s #success=1 #failure=8
    Readiness:  http-get http://:10902/-/ready delay=0s timeout=1s period=5s #success=1 #failure=20
    Environment:
      OBJSTORE_CONFIG:  <set to the key 'thanos.yaml' in secret 'thanos-objectstorage'>  Optional: false
      HOST_IP_ADDRESS:   (v1:status.hostIP)
    Mounts:
      /var/thanos/store from data (rw)
  Volumes:      <none>
Volume Claims:  <none>
Events:
  Type     Reason        Age                   From                    Message
  ----     ------        ----                  ----                    -------
  Warning  FailedCreate  8m41s (x19 over 30m)  statefulset-controller  create Pod thanos-store-0 in StatefulSet thanos-store failed error: Pod "thanos-store-0" is invalid: spec.containers[0].volumeMounts[0].name: Not found: "data"

Logic exists to ensure any passed volumeClaimTemplate is qualified to be used with thanos-store, if a template is passed: https://github.com/thanos-io/kube-thanos/blob/main/jsonnet/kube-thanos/kube-thanos-store.libsonnet#L28

This logic should be extended to include an emptyDir volume definition for "data" in the StatefulSet if no volumeClaimTemplate is passed.

[shubham-uipath]

@itspngu did you also tried any other thanos component without a pvc (may be with hostpath and emptydir), like receiver or ruler ? And can it work with non-retaining pvc's considering HA needs

[devattitanhq]

Am I following the right thread?
I get this error in the logs of the thanos-store-0 pod (in my Kind cluster):

Error from server (BadRequest): container "thanos-store" in pod "thanos-store-0" is waiting to start: CreateContainerConfigError

and the related event in the description reads:

  Warning  Failed     13s (x8 over 89s)  kubelet            Error: secret "thanos-objectstorage" not found

and in the stateful set:

    Environment:
      OBJSTORE_CONFIG:  <set to the key 'thanos.yaml' in secret 'thanos-objectstorage'>  Optional: false
      HOST_IP_ADDRESS:   (v1:status.hostIP)

I tried to add to it, but without success:

      volumes:
      - emptyDir: {}
        name: data

[shazib-summar]

@devattitanhq were you able to figure this out. I'm having the same issue.

[mgirod]

Just the indentation...

volumes:
  - emptyDir: {}
    name: data