failed to flush user; store put chunk: NoCredentialProviders

mrszop commented 1 year ago

Describe the bug I am running Loki/Promtail via Helm Charts von GCP/GKE:

Loki 2.6.1 Helm 1.8.11: https://bityl.co/Echb
Promtail 2.6.1 Helm 6.4.0: https://bityl.co/EaaJ

I noticed via Google Notification that my current billing is reaching its defined maximum and I was shocked why that is. In the billing console I was able to track down some "strange" Logging behaviour that has eaten up my whole account. I drilled deeped and found out that on all of my GKE clusters I have installed loki/promtail, I was getting a ton (7 million log entries per day) of following log entries:

level=error ts=2022-11-09T15:15:08.356391217Z caller=flush.go:146 org_id=fake msg="failed to flush user" err="store put chunk: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"

I saw via kubectl that the writer pods where running, but not ready:

NAME                                               READY   STATUS    RESTARTS   AGE                                                                                                                 
pod/loki-gateway-6558c495d4-qfd8p                  1/1     Running   0          4d5h                                                                                                                
pod/loki-grafana-agent-operator-79867d6656-577xs   1/1     Running   0          4d6h                                                                                                                
pod/loki-logs-898c6                                2/2     Running   0          4d6h                                                                                                                
pod/loki-logs-8xnfd                                2/2     Running   0          4d7h                                                                                                                
pod/loki-logs-c6dgs                                2/2     Running   0          4d6h                                                                                                                
pod/loki-logs-ccz64                                2/2     Running   0          4d6h                                                                                                                
pod/loki-logs-ffztb                                2/2     Running   0          4d5h                                                                                                                
pod/loki-logs-gqcwt                                2/2     Running   0          4d7h                                                                                                                
pod/loki-logs-qzf54                                2/2     Running   0          4d7h                                                                                                                
pod/loki-logs-w52jh                                2/2     Running   0          4d5h                                                                                                                
pod/loki-read-0                                    1/1     Running   0          4d5h                                                                                                                
pod/loki-read-1                                    1/1     Running   0          4d6h                                                                                                                
pod/loki-read-2                                    1/1     Running   0          4d7h                                                                                                                
pod/loki-write-0                                   0/1     Running   0          4d4h                                                                                                                
pod/loki-write-1                                   0/1     Running   0          4d6h                                                                                                                
pod/loki-write-2                                   0/1     Running   0          4d5h                                                                                                                
pod/promtail-2fqg7                                 1/1     Running   0          4d5h                                                                                                                
pod/promtail-6w4q4                                 1/1     Running   0          4d5h                                                                                                                
pod/promtail-d9zkv                                 1/1     Running   0          4d7h                                                                                                                
pod/promtail-gqlbr                                 1/1     Running   0          4d7h                                                                                                                
pod/promtail-jwdfk                                 1/1     Running   0          4d6h                                                                                                                
pod/promtail-n64gw                                 1/1     Running   0          4d6h                                                                                                                
pod/promtail-smb2b                                 1/1     Running   0          4d7h                                                                                                                
pod/promtail-z7lgw                                 1/1     Running   0          4d6h

There was no change from my side since days and Loki/Promtail was up and running fine. I was able to search logs via Grafana, no issues.

To Reproduce Steps to reproduce the behavior:

Install Loki via Helm Chart: https://bityl.co/Echb with following values.yaml:

loki:
  auth_enabled: false
  commonConfig:
    replication_factor: 3
monitoring:
  selfMonitoring:
    enabled: true
write:
  replicas: 3
  resources:
    limits:
      cpu: 600m
    requests:
      cpu: 300m
      memory: 384Mi
  persistence:
    storageClass: premium-rwo
    size: "50Gi"
read:
  replicas: 3
  autoscaling:
    enabled: true
    minReplicas: 3
    maxReplicas: 5
  resources:
    limits:
      cpu: 550m
      memory: 256Mi
    requests:
      cpu: 350m
      memory: 128Mi
  persistence:
    storageClass: premium-rwo
gateway:
  enabled: true
  replicas: 1
  verboseLogging: true
  autoscaling:
    enabled: true
    minReplicas: 1
    maxReplicas: 3
  resources:
    limits:
      cpu: 150m
      memory: 64Mi
    requests:
      cpu: 150m
      memory: 64Mi

Install promtail via Helm chart: https://bityl.co/EaaJ with following values.yaml:

resources:
  requests:
    cpu: 50m
    memory: 128Mi
  limits:
    cpu: 100m
    memory: 128Mi

Expected behavior

Environment:

GKE: Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.8-gke.1900", GitCommit:"79209257257c051b27df67c567755783eda93353", GitTreeState:"clean", BuildDate:"2022-07-15T09:23:51Z", GoVersion:"go1.17.11b7", Compiler:"gc", Platform:"linux/amd64"}
- Deployment tool: helm, helmfile, gitlab ci/cd

Questions

what does this log entry actually means and why does it need to be repeated like 7M times a day? I am not sure why aws.Config.CredentialsChainVerboseErrors actually is showing up, since I am not running on AWS or have configured anything but the values.yaml listed above
Is it possible to reduce the amount of loki log entry?

Sorry if I am showing a bit of unprofessional feelings here but I am a bit nervous about upcomming GCP bill produced by the unnecessary amount of loki logs.

Cheers

LarsBingBong commented 1 year ago

I experienced this when using a Loki component that could not use credentials being injected via the Helm extraEnvFrom key.

So maybe you need to configure your global storage to use credentials directly.

MohammedNoureldin commented 7 months ago

@mrszop have you figured it out? I have exactly the same issue. I am not using AWS at all, though I see this issue that mentioned AWS, which makes no sense.

kirill-lappo-dg7 commented 6 months ago

i also noticed that error message at startup

level=error ts=2024-03-05T08:20:46.798163605Z caller=reporter.go:205 msg="failed to delete corrupted cluster seed file, deleting it" err="NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors

I am using official helm chart

NAME                            CHART VERSION   APP VERSION     DESCRIPTION
grafana/loki                    5.42.0          2.9.3           Helm chart

upd might me connected with https://github.com/grafana/loki/issues/5941

grafana / loki

failed to flush user; store put chunk: NoCredentialProviders #7648