GCS Storage does not work w/Helm Chart

[ericmeadows]

Describe the bug I installed Loki using the 5.0.0 Helm Chart, using GCS as my storage.type, but it does not work.

As an aside, if feels like the documentation surrounding GCS is lacking.

To Reproduce Steps to reproduce the behavior:

1. Create a IAM Service Account in GCP (using Terraform) with the following configuration pieces (boilerplate removed)

resource "google_service_account" "loki" {
  ...
}

resource "google_storage_bucket_iam_binding" "loki-object-admin" {
  role   = "roles/storage.objectAdmin"
  ...
}

resource "google_storage_bucket_iam_binding" "loki-object-buckets-list" {
  role   = "roles/storage.legacyBucketReader"
  ...
}

resource "google_service_account_iam_binding" "loki" {
  role               = "roles/iam.workloadIdentityUser"

  members = [
    "serviceAccount:${var.project_id}.svc.id.goog[installedNamespace/loki-gcs-enabled]"
  ]
}

2. Create a values file that has the below configuration around storage;

serviceAccount:
  name: loki-gcs-enabled
  annotations:
    iam.gke.io/gcp-service-account: existingServiceAccount@ourProject.iam.gserviceaccount.com
storage:
  bucketNames:
    chunks: existingBucketForUs
    # ruler: existingBucketForUs
    # admin: existingBucketForUs
  type: gcs
  gcs:
    bucket_name: existingBucketForUs
    # service_account: <removed>
    chunkBufferSize: 0
    requestTimeout: "10s"
    enableHttp2: true

3. Tail the logs
4. See the following error

level=error ts=2023-04-07T00:43:11.733305605Z caller=flush.go:144 org_id=self-monitoring msg="failed to flush" err="failed to flush chunks: store put chunk: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors, num_chunks: 1, labels: {app_kubernetes_io_component=\"write\", app_kubernetes_io_instance=\"monitoring\", app_kubernetes_io_name=\"loki\", app_kubernetes_io_part_of=\"memberlist\", cluster=\"monitoring-loki\", container=\"loki\", controller_revision_hash=\"loki-write-f7dc4644d\", filename=\"/var/log/pods/monitoring_loki-write-1_44539599-7ee4-40ed-b15d-cf7eb62394bb/loki/0.log\", job=\"monitoring/loki-write\", namespace=\"monitoring\", pod=\"loki-write-1\", statefulset_kubernetes_io_pod_name=\"loki-write-1\", stream=\"stderr\"}"

Expected behavior Loki to spin up and store objects in our GCS bucket.

Environment:

• Infrastructure: Kubernetes
• Deployment tool: Helm

Screenshots, Promtail config, or terminal output

level=error ts=2023-04-07T01:03:06.364053187Z caller=flush.go:144 org_id=self-monitoring msg="failed to flush" err="failed to flush chunks: store put chunk: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors, num_chunks: 1, labels: {app_kubernetes_io_component=\"canary\", app_kubernetes_io_instance=\"monitoring\", app_kubernetes_io_name=\"loki\", cluster=\"monitoring-loki\", container=\"loki-canary\", controller_revision_hash=\"6dbd8cdb8c\", filename=\"/var/log/pods/monitoring_loki-canary-lr2wj_30c88ab5-70d0-4cb3-8799-b02e95183319/loki-canary/0.log\", job=\"monitoring/loki-canary\", namespace=\"monitoring\", pod=\"loki-canary-lr2wj\", pod_template_generation=\"1\", stream=\"stderr\"}"
level=error ts=2023-04-07T01:03:06.364077017Z caller=flush.go:144 org_id=self-monitoring msg="failed to flush" err="failed to flush chunks: store put chunk: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors, num_chunks: 1, labels: {app_kubernetes_io_component=\"gateway\", app_kubernetes_io_instance=\"monitoring\", app_kubernetes_io_name=\"loki\", cluster=\"monitoring-loki\", container=\"nginx\", filename=\"/var/log/pods/monitoring_loki-gateway-5c9c45879d-wkm6z_c4113bfc-a5e4-4f10-9abf-a98a38818b88/nginx/0.log\", job=\"monitoring/loki-gateway\", namespace=\"monitoring\", pod=\"loki-gateway-5c9c45879d-wkm6z\", pod_template_hash=\"5c9c45879d\", stream=\"stderr\"}"
level=error ts=2023-04-07T01:03:06.364720077Z caller=flush.go:144 org_id=self-monitoring msg="failed to flush" err="failed to flush chunks: store put chunk: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors, num_chunks: 1, labels: {app_kubernetes_io_component=\"canary\", app_kubernetes_io_instance=\"monitoring\", app_kubernetes_io_name=\"loki\", cluster=\"monitoring-loki\", container=\"loki-canary\", controller_revision_hash=\"6dbd8cdb8c\", filename=\"/var/log/pods/monitoring_loki-canary-bnswf_3fca9086-2d4a-4bcc-8b3b-308d072cbcba/loki-canary/0.log\", job=\"monitoring/loki-canary\", namespace=\"monitoring\", pod=\"loki-canary-bnswf\", pod_template_generation=\"1\", stream=\"stdout\"}"
level=error ts=2023-04-07T01:03:06.365137427Z caller=flush.go:144 org_id=self-monitoring msg="failed to flush" err="failed to flush chunks: store put chunk: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors, num_chunks: 1, labels: {app_kubernetes_io_component=\"read\", app_kubernetes_io_instance=\"monitoring\", app_kubernetes_io_name=\"loki\", app_kubernetes_io_part_of=\"memberlist\", cluster=\"monitoring-loki\", container=\"loki\", controller_revision_hash=\"loki-read-7b65cf5677\", filename=\"/var/log/pods/monitoring_loki-read-2_cec67367-46f4-4d44-bb22-1f9b3fe704e7/loki/0.log\", job=\"monitoring/loki-read\", namespace=\"monitoring\", pod=\"loki-read-2\", statefulset_kubernetes_io_pod_name=\"loki-read-2\", stream=\"stderr\"}"
level=error ts=2023-04-07T01:03:06.365544097Z caller=flush.go:144 org_id=self-monitoring msg="failed to flush" err="failed to flush chunks: store put chunk: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors, num_chunks: 1, labels: {app_kubernetes_io_component=\"read\", app_kubernetes_io_instance=\"monitoring\", app_kubernetes_io_name=\"loki\", app_kubernetes_io_part_of=\"memberlist\", cluster=\"monitoring-loki\", container=\"loki\", controller_revision_hash=\"loki-read-7b65cf5677\", filename=\"/var/log/pods/monitoring_loki-read-0_38acb48f-ba3c-4324-9148-3c286d11bc53/loki/0.log\", job=\"monitoring/loki-read\", namespace=\"monitoring\", pod=\"loki-read-0\", statefulset_kubernetes_io_pod_name=\"loki-read-0\", stream=\"stderr\"}"
level=info ts=2023-04-07T01:03:06.365589687Z caller=flush.go:168 msg="flushing stream" user=self-monitoring fp=d8e196886a0c1134 immediate=false num_chunks=1 labels="{app_kubernetes_io_component=\"write\", app_kubernetes_io_instance=\"monitoring\", app_kubernetes_io_name=\"loki\", app_kubernetes_io_part_of=\"memberlist\", cluster=\"monitoring-loki\", container=\"loki\", controller_revision_hash=\"loki-write-f7dc4644d\", filename=\"/var/log/pods/monitoring_loki-write-1_44539599-7ee4-40ed-b15d-cf7eb62394bb/loki/0.log\", job=\"monitoring/loki-write\", namespace=\"monitoring\", pod=\"loki-write-1\", statefulset_kubernetes_io_pod_name=\"loki-write-1\", stream=\"stderr\"}"
level=error ts=2023-04-07T01:03:06.371441136Z caller=flush.go:144 org_id=self-monitoring msg="failed to flush" err="failed to flush chunks: store put chunk: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors, num_chunks: 1, labels: {app_kubernetes_io_component=\"write\", app_kubernetes_io_instance=\"monitoring\", app_kubernetes_io_name=\"loki\", app_kubernetes_io_part_of=\"memberlist\", cluster=\"monitoring-loki\", container=\"loki\", controller_revision_hash=\"loki-write-f7dc4644d\", filename=\"/var/log/pods/monitoring_loki-write-1_44539599-7ee4-40ed-b15d-cf7eb62394bb/loki/0.log\", job=\"monitoring/loki-write\", namespace=\"monitoring\", pod=\"loki-write-1\", statefulset_kubernetes_io_pod_name=\"loki-write-1\", stream=\"stderr\"}"
level=info ts=2023-04-07T01:03:06.371482176Z caller=flush.go:168 msg="flushing stream" user=self-monitoring fp=2628f494230279f4 immediate=false num_chunks=1 labels="{app_kubernetes_io_component=\"canary\", app_kubernetes_io_instance=\"monitoring\", app_kubernetes_io_name=\"loki\", cluster=\"monitoring-loki\", container=\"loki-canary\", controller_revision_hash=\"6dbd8cdb8c\", filename=\"/var/log/pods/monitoring_loki-canary-9w66v_8ba9d453-f40f-465c-b185-f8fc12905751/loki-canary/0.log\", job=\"monitoring/loki-canary\", namespace=\"monitoring\", pod=\"loki-canary-9w66v\", pod_template_generation=\"1\", stream=\"stderr\"}"
level=error ts=2023-04-07T01:03:06.372185286Z caller=flush.go:144 org_id=self-monitoring msg="failed to flush" err="failed to flush chunks: store put chunk: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors, num_chunks: 1, labels: {app_kubernetes_io_component=\"canary\", app_kubernetes_io_instance=\"monitoring\", app_kubernetes_io_name=\"loki\", cluster=\"monitoring-loki\", container=\"loki-canary\", controller_revision_hash=\"6dbd8cdb8c\", filename=\"/var/log/pods/monitoring_loki-canary-9w66v_8ba9d453-f40f-465c-b185-f8fc12905751/loki-canary/0.log\", job=\"monitoring/loki-canary\", namespace=\"monitoring\", pod=\"loki-canary-9w66v\", pod_template_generation=\"1\", stream=\"stderr\"}"

[AgrimPrasad]

We're also hitting this issue now where GCS storage doesn't work despite configuring it with the helm chart for loki.

loki-write-0 loki level=error ts=2023-05-05T04:20:32.131662407Z caller=flush.go:144 org_id=fake msg="failed to flush" err="failed to flush chunks: store put chunk: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors, num_chunks: 1, labels: {app=\"host-scanner\", container=\"host-sensor\", filename=\"/var/log/pods/kubescape-host-scanner_host-scanner-knpc7_1d9d6c80-9edf-4011-9e6f-1d0cf^Cloki-write-0 loki level=error ts=2023-05-05T04:20:32.131858327Z caller=flush.go:144 org_id=fake msg="failed to flush" err="failed to flush chunks: store put chunk: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors, num_chunks: 1, labels: {app=\"app-name-redacted\", container=\"cloudsql-proxy\", filename=\"/var/log/pods/default_app-name-redacted-55d5f987cd-kd2st_5c0a8d7d-1abe-4b8a-bbcf-e38439c68828/cloudsql-proxy/0.log\", instance=\"app-name-redacted\", job=\"default/app-name-redacted\", namespace=\"default\", node_name=\"gke-dev-t2d-preemptible--badd0736-g2f5\", pod=\"app-name-redacted-55d5f987cd-kd2st\", stream=\"stderr\"}"

[jdsdc]

We're also hitting this issue now where GCS storage doesn't work despite configuring it with the helm chart for loki.

loki-write-0 loki level=error ts=2023-05-05T04:20:32.131662407Z caller=flush.go:144 org_id=fake msg="failed to flush" err="failed to flush chunks: store put chunk: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors, num_chunks: 1, labels: {app=\"host-scanner\", container=\"host-sensor\", filename=\"/var/log/pods/kubescape-host-scanner_host-scanner-knpc7_1d9d6c80-9edf-4011-9e6f-1d0cf^Cloki-write-0 loki level=error ts=2023-05-05T04:20:32.131858327Z caller=flush.go:144 org_id=fake msg="failed to flush" err="failed to flush chunks: store put chunk: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors, num_chunks: 1, labels: {app=\"app-name-redacted\", container=\"cloudsql-proxy\", filename=\"/var/log/pods/default_app-name-redacted-55d5f987cd-kd2st_5c0a8d7d-1abe-4b8a-bbcf-e38439c68828/cloudsql-proxy/0.log\", instance=\"app-name-redacted\", job=\"default/app-name-redacted\", namespace=\"default\", node_name=\"gke-dev-t2d-preemptible--badd0736-g2f5\", pod=\"app-name-redacted-55d5f987cd-kd2st\", stream=\"stderr\"}"

Did you find a solution? @AgrimPrasad

Having the same issue with azure storage

[AgrimPrasad]

We updated to latest grafana and loki versions and the gcs bug was fixed there

[medykn-git]

We updated to latest grafana and loki versions and the gcs bug was fixed there

Can you please share the configuration you've applied, and which version (Loki + Loki helm chart) works ?