Loki azure blob storage not storing the persistent logs

jomonr commented 3 months ago

I have deployed Loki in azure container app with azure blob storage. In my local machine, I was able to deploy with local config and is working, but when i deployed to azure container app with blob storage, its not getting persistent. I am loosing the logs when i restart the container app.

below is the loki yaml I am using.

auth_enabled: false

server:
  http_listen_port: 3100

common:
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2022-06-25
      index:
        period: 24h
        prefix: index_
      object_store: azure
      schema: v13
      store: boltdb-shipper

storage_config:
  azure:
    use_managed_identity: false
    account_name: <azure_blob_account_name>
    account_key: <azure_blob_account_key>
    container_name: lokidata
    request_timeout: 0
  boltdb_shipper:
    active_index_directory: /loki/boltdb-shipper-active
    cache_location: /loki/boltdb-shipper-cache
    cache_ttl: 24h
    shared_store: azure
  filesystem:
    directory: /loki/chunks

ruler:
  alertmanager_url: http://localhost:9093

compactor:
  working_directory: /loki/compactor

ingester:
  wal:
    dir: /loki/wal

I can see in azure blob storage the seed file is created , but nothing else.

I am using the latest loki version.

Currently I am stuck with this, please let me know if I am missing something , Any help will be much appreciated.

You may submit questions in any of the following ways:

NOTE: All questions submitted as issues will be closed.

PedemonteGiacomo commented 2 months ago

I'm having the same with this configuration

auth_enabled: false

server:
  http_listen_port: {{ .Values.loki.ports.http }}
  grpc_listen_port: {{ .Values.loki.ports.grpc }}

common:
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory

schema_config:
  configs:
  - from: "2020-12-11"
    index:
      period: 24h
      prefix: index_
    object_store: azure  
    schema: v13
    store: tsdb

storage_config:
  azure:
    account_name: {{ .Values.storage.azure.accountName }}
    account_key: {{ .Values.storage.azure.accountKey }}
    container_name: {{ .Values.storage.azure.containerName }}
    use_managed_identity: false
    endpoint_suffix: blob.core.windows.net
  tsdb_shipper:
    active_index_directory: /tmp/loki/index
    cache_location: /tmp/loki/index_cache
    cache_ttl: 24h

cscn commented 2 months ago

Are you using the latest helm chart? If so, a lot of the storage_config options have moved to storage instead, and have changed to camel-case instead of snake-case.

For example, I think the above should now be (as of Grafana Loki v2.6.x):

...

storage:
  type: azure
  azure:
    accountName: {{ .Values.storage.azure.accountName }}
    accountKey: { .Values.storage.azure.accountKey }}
    useManagedIdentity: false
  bucketNames:
    chunks: {{ .Values.storage.azure.containerName }}
    ruler: {{ .Values.storage.azure.containerName }}
    admin: {{ .Values.storage.azure.containerName }}
storage_config:
  azure:
    container_name: {{ .Values.storage.azure.containerName }} # Nothing wrote for me unless I added this here as well
  tsdb_shipper:
    active_index_directory: /tmp/loki/index
    cache_location: /tmp/loki/index_cache
    cache_ttl: 24h
...

Here are some resources that helped me:

JStickler commented 3 weeks ago

Configuration questions have a better chance of being answered if you ask them on the community forums.

grafana / loki

Loki azure blob storage not storing the persistent logs #13850