grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
24.04k stars 3.47k forks source link

container_name in azure_storage_config is not used #9483

Open ivanmagdic opened 1 year ago

ivanmagdic commented 1 year ago

Describe the bug When defining 'container_name' in 'azure_storage_config' a provided name is not used, and instead a 'chunks' name is used for the container name. It seems to be inherited from 'loki.storage.bucketNames.chunks' because if you change that value, for example, if we set it to 'logs', that container name ('logs') is used in config.yaml.

To Reproduce Steps to reproduce the behavior:

  1. Deploy and start Loki with Helm

Expected behavior A 'container_name' that is defined in 'azure_storage_config ' property should be used instead of 'loki.storage.bucketNames.chunks' to define container name.

Environment:

Screenshots, Promtail config, or terminal output When using following Helm'values.yaml':

loki:
  auth_enabled: false
  server:
    http_listen_port: 3100
  commonConfig:
    replication_factor: 1
  storage:
    type: azure
    azure:
      accountName: REDACTED
      accountKey: REDACTED
      container_name: logs

loki 'config.yaml' is following:

auth_enabled: false
common:
  compactor_address: 'loki-backend'
  path_prefix: /var/loki
  replication_factor: 1
  storage:
    azure:
      account_key: REDACTED
      account_name: REDACTED
      container_name: chunks
      use_federated_token: false
      use_managed_identity: false
...

If we change the Helm 'values.yaml':

loki:
  auth_enabled: false
  server:
    http_listen_port: 3100
  commonConfig:
    replication_factor: 1
  storage:
    bucketNames:
      chunks: logs <--- This values is used
    type: azure
    azure:
      accountName: REDACTED
      accountKey: REDACTED
      container_name: loki

the 'config.yaml' will be like following:

auth_enabled: false
common:
  compactor_address: 'loki-backend'
  path_prefix: /var/loki
  replication_factor: 1
  storage:
    azure:
      account_key: REDACTED
      account_name: REDACTED
      container_name: logs
      use_federated_token: false
      use_managed_identity: false
...
kirill-lappo-dg7 commented 5 months ago

It would be great to describe that in documentation, per each storage provider. For example, in Azure you need to create a container per each bucket name.

Right now you can only guess that by accident or analyzing urls that Loki backend logs into stdout

stieler-it commented 5 months ago

The tricky part is that the Helm chart wraps the storage config, so it is not possible to just use the Loki docs on storage config. So as @kirll-lappo-dg7 suggested, if this is intentional, maybe just some docs are missing.

The implementation for Azure is here https://github.com/grafana/loki/blob/v3.0.0/production/helm/loki/templates/_helpers.tpl#L280 and I saw that the same is valid for S3 https://github.com/grafana/loki/blob/v3.0.0/production/helm/loki/templates/_helpers.tpl#L221 - Helm value bucket_name / bucketName are not required here (as opposed to configuring Loki without Helm).

kirill-lappo-dg7 commented 5 months ago

The tricky part is that the Helm chart wraps the storage config, so it is not possible to just use the Loki docs on storage config. So as @kirll-lappo-dg7 suggested, if this is intentional, maybe just some docs are missing.

The implementation for Azure is here https://github.com/grafana/loki/blob/v3.0.0/production/helm/loki/templates/_helpers.tpl#L280 and I saw that the same is valid for S3 https://github.com/grafana/loki/blob/v3.0.0/production/helm/loki/templates/_helpers.tpl#L221 - Helm value bucket_name / bucketName are not required here (as opposed to configuring Loki without Helm).

Still you may specify more than one backet names, which will be used as container name as well (as far as I understood from error messages in Loki backend)