Open exb-atix opened 10 months ago
im seeing this as well but on 2.9.1
Sorry im not using loki helm but i figured it out.
The config must have the region set here when using IRSA
common:
compactor_address: 'loki'
path_prefix: /var/loki
replication_factor: 2
storage:
s3:
bucketnames: {{ .Values.s3_bucket }}
region: {{ .Values.region }}
Mines working now. Env vars did not matter for whatever reason.
Sorry im not using loki helm but i figured it out.
The config must have the region set here when using IRSA
common: compactor_address: 'loki' path_prefix: /var/loki replication_factor: 2 storage: s3: bucketnames: {{ .Values.s3_bucket }} region: {{ .Values.region }}
Mines working now. Env vars did not matter for whatever reason.
Thank you for your response. I tried your suggestion and put the storage block into the commonConfig
block in my config, but unfortunately the issue is still the same.
I have the same issue
I'm facing the same issue any update on this?
I had the same error message, deploying via helm and loki as singleBinary. After adding the list-element "ruler: BUCKET_NAME" it disapeared
# values.yaml
loki:
..
storage:
bucketNames:
chunks: BUCKET_NAME
ruler: BUCKET_NAME
type: s3
s3:
s3: s3://BUCKET_NAME
region: "eu-central-1"
accessKeyId: "${GRAFANA_LOKI_S3_ACCESKEYID}"
secretAccessKey: "${GRAFANA_LOKI_S3_SECRETACCESSKEY}"
s3ForcePathStyle: false
insecure: false
https://grafana.com/docs/loki/latest/setup/install/helm/install-monolithic/
Which makes sense, since the helm chart's _helpers.tpl is looking for $.Values.loki.storage.bucketNames.ruler
https://github.com/grafana/loki/blob/main/production/helm/loki/templates/_helpers.tpl#L342
hello 0xdnL, thank you for this hint. I was now able to test your suggestion, but unfortunately the error persists. This is my change that i tried (among several variations):
commonConfig:
path_prefix: /var/loki
replication_factor: 3
storage:
bucketNames:
ruler: {{ .Values.loki.bucket_name }}
chunks: {{ .Values.loki.bucket_name }}
type: s3
s3:
s3: {{ .Values.loki.bucket_name }}
region: {{ .Values.aws.region }}
s3forcepathstyle: false
Can someone from grafana add a definitive working exemple values.yaml in exemple directory for distributed loki with S3 backend using IRSA ?
small update: we upgraded to loki 3.0.0
via helm chart version 6.3.3
and the error still persists.
Can confirm this is still the case as of Chart v6.6.4:
init compactor: failed to init delete store: failed to get s3 object: WebIdentityErr: failed to retrieve credentials
caused by: RequestError: send request failed
caused by: Post "https://sts.dummy.amazonaws.com/": 3 errors occurred:
* dial tcp: lookup sts.dummy.amazonaws.com on 172.20.0.10:53: no such host
* dial tcp: lookup sts.dummy.amazonaws.com on 172.20.0.10:53: no such host
* dial tcp: lookup sts.dummy.amazonaws.com on 172.20.0.10:53: no such host
This is still an issue in chart v 6.12.0
Describe the bug AWS region is not taken into account at least for
loki-backend
pods when trying to access AWS STS. This is throwing continuously error messages (see log in output section). Apart from these error messages, everything is working as expected though. After looking into the code (as non-native Go speaker) the culprit seems to lie around the lines 224-232 and 245-247 ofs3_storage_client.go
where the region should be set into the s3Config object.To Reproduce Steps to reproduce the behavior:
loki-backend
podExpected behavior The used endpoint is not filled with
dummy
region and thus doesn't throw an error.Environment:
Screenshots, Promtail config, or terminal output
Loki log:
Loki helm values:
Screenshot of the applied env vars: