Default buckets are not available when install the helmchart with minio

[kaflake]

I try to install the loki-helm-chart with its integrated minio (over flux) with this values.yaml:

loki:
  serviceMonitor:
    enabled: true
  auth_enabled: false
minio:
  enabled: true

Now the loki-backend will not start with this error an error that the buckets in minio are not present (see at the bottom). The buckets will be installed normaly over an post-script. This script will never run here, cause the loki-backend will ready and so helm install is never completed -> deadlock. Is there a solution to fix this with the chart? Or have I to write an own script to add the buckets?

level=error ts=2023-06-05T15:16:20.320156891Z caller=log.go:171 msg="error running loki" err="NoSuchBucket: The specified bucket does not exist\n\tstatus code: 404, request id: 1765CC38E9022C3E, host id: \nfailed to get s3 object\ngithub.com/grafana/loki/pkg/storage/chunk/client/aws.(*S3ObjectClient).GetObject\n\t/src/loki/pkg/storage/chunk/client/aws/s3_storage_client.go:396\ngithub.com/grafana/loki/pkg/storage/stores/indexshipper/storage.prefixedObjectClient.GetObject\n\t/src/loki/pkg/storage/stores/indexshipper/storage/prefixed_object_client.go:25\ngithub.com/grafana/loki/pkg/storage/stores/indexshipper/storage.(*indexStorageClient).GetFile\n\t/src/loki/pkg/storage/stores/indexshipper/storage/client.go:128\ngithub.com/grafana/loki/pkg/storage/stores/indexshipper/compactor/deletion.(*deleteRequestsTable).init.func1\n\t/src/loki/pkg/storage/stores/indexshipper/compactor/deletion/delete_requests_table.go:70\ngithub.com/grafana/loki/pkg/storage/stores/indexshipper/storage.DownloadFileFromStorage\n\t/src/loki/pkg/storage/stores/indexshipper/storage/util.go:48\ngithub.com/grafana/loki/pkg/storage/stores/indexshipper/compactor/deletion.(*deleteRequestsTable).init\n\t/src/loki/pkg/storage/stores/indexshipper/compactor/deletion/delete_requests_table.go:68\ngithub.com/grafana/loki/pkg/storage/stores/indexshipper/compactor/deletion.newDeleteRequestsTable\n\t/src/loki/pkg/storage/stores/indexshipper/compactor/deletion/delete_requests_table.go:50\ngithub.com/grafana/loki/pkg/storage/stores/indexshipper/compactor/deletion.NewDeleteStore\n\t/src/loki/pkg/storage/stores/indexshipper/compactor/deletion/delete_requests_store.go:63\ngithub.com/grafana/loki/pkg/storage/stores/indexshipper/compactor.(*Compactor).initDeletes\n\t/src/loki/pkg/storage/stores/indexshipper/compactor/compactor.go:281\ngithub.com/grafana/loki/pkg/storage/stores/indexshipper/compactor.(*Compactor).init\n\t/src/loki/pkg/storage/stores/indexshipper/compactor/compactor.go:265\ngithub.com/grafana/loki/pkg/storage/stores/indexshipper/compactor.NewCompactor\n\t/src/loki/pkg/storage/stores/indexshipper/compactor/compactor.go:235\ngithub.com/grafana/loki/pkg/loki.(*Loki).initCompactor\n\t/src/loki/pkg/loki/modules.go:1037\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:120\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:92\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:457\nmain.main\n\t/src/loki/cmd/loki/main.go:110\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598\nerror initialising module: compactor\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:122\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:92\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:457\nmain.main\n\t/src/loki/cmd/loki/main.go:110\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598"

Greats

[naviat]

@kaflake do you have any actions to bypass this?

[Kampe]

Seeing the same issue with the latest chart version 5.19.0

[kaflake]

I have add the post-script job manualy under extraObjects in the helm chart.

Take the code from post-job.yaml and modify it for my needs.

[MikaelElkiaer]

I am also experiencing this issue.

@kaflake Nice advice about the post-job.

I basically ran

helm show values grafana/loki | yq '.minio' > values.yaml
helm template minio/minio --name-template loki --namespace monitoring \
  --output-dir . --values values.yaml --version 4.0.15

and copied the contents of ./minio/post-install-create-bucket-job.yaml in as extraObjects in Loki values - removing the Helm hook annotations. Note that I locked the Minio chart version to the one specified by the Loki chart, and that I am grabbing Minio values from the latest Loki chart.

[adberger]

Note: This also seems to be a problem when installing loki via ArgoCD, since the Sync will never succeed and therefore the post-install hook will not be run.

https://argo-cd.readthedocs.io/en/stable/user-guide/helm/#helm-hooks: helm.sh/hook: post-install Supported as equivalent to argocd.argoproj.io/hook: PostSync.

https://argo-cd.readthedocs.io/en/stable/user-guide/resource_hooks/#usage PostSync Executes after all Sync hooks completed and were successful, a successful application, and all resources in a Healthy state.

Workaround: Scale failing statefulsets to 0, wait for ArgoCD Application sync to succeed, minio Jobs will run and then scale failing statefulsets again.

A better workaround might be like in mimir-distributed Helm Chart: https://github.com/grafana/mimir/blob/7239211aa283a517b300f543b94aeccacd2fcc61/operations/helm/charts/mimir-distributed/templates/minio/create-bucket-job.yaml

[sunidhi271]

For me it worked only after adding the bucket details that is there in default value file, after disabling the compactor and removing compactor configurations-

[TheMatrix97]

Facing the same issue here when deploying loki helm using argocd. In our case, we are deploying the chart using kustomization with the helmchartinflator

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: monitoring
helmCharts:
- name: loki
  repo: https://grafana.github.io/helm-charts/
  releaseName: loki
  namespace: core
  version: 6.6.6
  includeCRDs: true
  valuesFile: values-loki.yaml

patches:
  - path: patch-loki-jobs.yaml
    target:
      kind: Job

To make it work, we had to patch all loki jobs with the argocd annotations, overriding the default helm hooks. Then, argocd is triggering all jobs

patch-loki-jobs.yaml

apiVersion: batch/v1
kind: Job
metadata:
  annotations:
    argocd.argoproj.io/hook: Sync
    argocd.argoproj.io/sync-wave: "0"
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation,HookSucceeded
  name: none