How to configure aws s3 credentials securely for accessing aws storage object in loki-configuration

[vermaprateek695]

Hi All ,

In our approach we are defining the entire aws s3 configuration in a secret and creating environment variables in the loki config to access the secret containing AWS s3 details , but when trying to deploy and access the aws secret using env section defined in loki configuration

extraArgs: config.expand-env: true

env:

• name: S3_ACCESS_KEY_id valueFrom: secretKeyRef: name: s3-credentials key: access_key_id

• name: S3_SECRET_ACCESS_KEY valueFrom: secretKeyRef: name: s3-credentials key: secret_access_key

  aws: s3: bucketnames: mybucketnames access_key_id: ${S3_ACCESS_KEY_id} secret_access_key: ${S3_SECRET_ACCESS_KEY}

But we are getting below error

level=error ts=2023-02-21T06:31:04.487300046Z caller=flush.go:146 org_id=acc msg="failed to flush user" err="store put chunk: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.\n\tstatus code: 403,

Validated the access_key_id and secret_access_key , values looks good but still unable to access it.

Any leads will be appreciated for fixing this issue or maybe another workaround.

Thanks !

[jeschkies]

How are you deploying? Depending on your environment you could use an IAM role.

[jonas-tm]

I am also interested in how to configure this properly (unrelated to the cloud provider). Seems like a lot of people are having the same problem here. To my knowledge it is bad practice to put secrets into helm values instead secret references should be used.

[vermaprateek695]

How are you deploying? Depending on your environment you could use an IAM role.

Hi @jeschkies

We are deploying using helm chart and its a dev environment where we are using it.. can you let me know how we can implement AWS IAM role for the same ?

Any ideas and suggestions are welcomed .

Regards, Prateek

[Pela647]

@vermaprateek695 , please use IRSA and follow everything here: https://github.com/grafana/loki/issues/8152

[mrwormhole]

current problem seems to be confusion around where to put extraArgs: config.expand-env: true and helm charts not supporting secretKeyRef

sorry to jump but not everyone uses AWS IAM & SA in their clusters @Pela647

[PaoloC68]

Please clarify where extraArgs: config.expand-env: true belongs! Do I have to set also extraEnv: or extraEnvFrom: or it is enough one of them?

[davidusken]

@PaoloC68 just managed to set this up, you will need both for each pod. I am using a Helm chart to deploy but this is how I set it up:

...
  write:
    replicas: 1

    # Disable podAntiAffinity because of running on single-node
    affinity: {}
    extraArgs:
      - '-config.expand-env=true'
    extraEnv:
      - name: S3_LOKI_ACCESS_KEY_ID
        valueFrom:
          secretKeyRef:
            name: loki-bucket-secret
            key: S3_LOKI_ACCESS_KEY_ID
      - name: S3_LOKI_SECRET_ACCESS_KEY
        valueFrom:
          secretKeyRef:
            name: loki-bucket-secret
            key: S3_LOKI_SECRET_ACCESS_KEY

  read:
    replicas: 1
    persistence:
      enableStatefulSetAutoDeletePVC: false

    # Disable podAntiAffinity because of running on single-node
    affinity: {}
    extraArgs:
      - '-config.expand-env=true'
    extraEnv:
      - name: S3_LOKI_ACCESS_KEY_ID
        valueFrom:
          secretKeyRef:
            name: loki-bucket-secret
            key: S3_LOKI_ACCESS_KEY_ID
      - name: S3_LOKI_SECRET_ACCESS_KEY
        valueFrom:
          secretKeyRef:
            name: loki-bucket-secret
            key: S3_LOKI_SECRET_ACCESS_KEY
...

[esmaeilzadehayub]

I am using a Helm chart to deploy Loki, and it is working fine

backend:
  extraArgs:
    - '-config.expand-env=true'
  extraEnv:
    - name: S3_LOKI_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          name: loki-bucket-secret
          key: S3_LOKI_ACCESS_KEY_ID
    - name: S3_LOKI_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: loki-bucket-secret
          key: S3_LOKI_SECRET_ACCESS_KEY

write:
  extraArgs:
    - '-config.expand-env=true'
  extraEnv:
    - name: S3_LOKI_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          name: loki-bucket-secret
          key: S3_LOKI_ACCESS_KEY_ID
    - name: S3_LOKI_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: loki-bucket-secret
          key: S3_LOKI_SECRET_ACCESS_KEY

read:
  extraArgs:
    - '-config.expand-env=true'
  extraEnv:
    - name: S3_LOKI_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          name: loki-bucket-secret
          key: S3_LOKI_ACCESS_KEY_ID
    - name: S3_LOKI_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: loki-bucket-secret
          key: S3_LOKI_SECRET_ACCESS_KEY

[jeschkies]

Maybe we should make it simple. Please note, if you are running in AWS you don't d to pass the credentials at all.

[esmaeilzadehayub]

Close this issue, it is a solution https://github.com/grafana/loki/issues/8572#issuecomment-1582831358

[tomaaron]

Maybe we should make it simple. Please note, if you are running in AWS you don't d to pass the credentials at all.

I agree. This should be simpler. As far as I read the docs from the values there should be an option to set global.extraEnvFrom but found nowhere a helper or reference to this.

[david-nano]

Close this issue, it is a solution #8572 (comment)

Sorry for this update, but even this practice didn't worked on "backend" deployment, log:

level=error ts=2023-07-24T10:44:48.586258436Z caller=cached_client.go:65 msg="failed to build cache" err="NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"
level=error ts=2023-07-24T10:44:48.586313335Z caller=index_set.go:285 table-name=loki_index_19562 user-id=fake msg="sync failed, retrying it" err="NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors"
ts=2023-07-24T10:44:48.586346335Z caller=spanlogger.go:85 level=info msg="building index list cache"

Full log: loki.log

@esmaeilzadehayub Where did you find the environment variables to use? I see the are mapped, but backend refuse to use them.

[WoodyWoodsta]

More confusion here. The dots that are not being joined are that you need to add tokens in the config which '-config.expand-env=true' will expand, as per: https://grafana.com/docs/loki/latest/configuration/#use-environment-variables-in-the-configuration

So you need:

loki:
  s3:
    # ... s3 config
    accessKeyId: ${S3_LOKI_ACCESS_KEY_ID}
    secretAccessKey: ${S3_LOKI_SECRET_ACCESS_KEY}

As well as what has been mentioned already, where the values for the above tokens will come from:

backend:
  extraArgs:
    - '-config.expand-env=true'
  extraEnv:
    - name: S3_LOKI_ACCESS_KEY_ID
      valueFrom:
        secretKeyRef:
          name: loki-bucket-secret
          key: S3_LOKI_ACCESS_KEY_ID
    - name: S3_LOKI_SECRET_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: loki-bucket-secret
          key: S3_LOKI_SECRET_ACCESS_KEY

# Repeated for `write` and `read`

[david-nano]

@WoodyWoodsta it's answer we saw already by @davidusken https://github.com/grafana/loki/issues/8572#issuecomment-1543652661

[WoodyWoodsta]

@WoodyWoodsta it's answer we saw already by @davidusken

https://github.com/grafana/loki/issues/8572#issuecomment-1543652661

No it's not. They only mention the second part. It was unclear whether the component reads in the environment variables directly or uses them to expand the config further. I was just trying to string the two together in one comment.

[matschaffer-roblox]

Thanks @WoodyWoodsta - I got confused by that exact point and seeing your complete example helped clear up if S3_LOKI_ACCESS_KEY_ID was some sort of "special" env var or not (answer: it's not).

[jhr1972]

For those in need to debug the env-expansion, it might be useful to add the print-config statement as it dumps you all the variables finally processed.

  extraArgs:
    - --config.expand-env=true
    - --log.level=debug
    - --print-config-stderr

[Angel0r]

I had similar problem #12218. Since I use Kubernetes Secrets I solved it by using extraEnvFrom instead of extraEnv.

loki:
  storage:
    type: s3
    s3:
      s3: "s3://test-loki"
      endpoint: "test.se-grid.com/"
      region: eu-north-1
      secretAccessKey: "${secretAccessKey}"
      accessKeyId: "${accessKeyId}"
      signatureVersion: null
      s3ForcePathStyle: true
      insecure: false
      http_config: {}

# Configuration for the backend pod(s)
backend:
  # -- Number of replicas for the backend
  replicas: 3
  extraArgs: ["-config.expand-env=true"]
  extraEnvFrom: ["loki-test-credentials"]

[Angel0r]

My issue is solved here #12218. Hope it helps.