[loki-distributed] error running loki" err="mkdir /var/loki: read-only file system\nerror"

[qdupuy]

Hello 🖖

when I try to deploy table-manager I get the following error :

k -n monitoring logs -f loki-table-manager-55c4b5cc4c-xwnzp
level=warn ts=2021-10-11T15:34:14.155064403Z caller=store.go:91 msg="running with DEPRECATED flag -store.max-look-back-period, use -querier.max-query-lookback instead."
level=info ts=2021-10-11T15:34:14.157149941Z caller=main.go:129 msg="Starting Loki" version="(version=, branch=, revision=)"
level=info ts=2021-10-11T15:34:14.157870211Z caller=server.go:239 http=[::]:3100 grpc=[::]:9095 msg="server listening on addresses"
level=error ts=2021-10-11T15:34:14.15831012Z caller=log.go:106 msg="error running loki" err="mkdir /var/loki: read-only file system\nerror initialising module: table-manager\ngithub.com/cortexproject/cortex/pkg/util/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/cortexproject/cortex/pkg/util/modules/modules.go:105\ngithub.com/cortexproject/cortex/pkg/util/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/cortexproject/cortex/pkg/util/modules/modules.go:75\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:241\nmain.main\n\t/src/loki/cmd/loki/main.go:131\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:225\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371"

Here is my loki config file :

  config: |
    auth_enabled: false

    server:
      http_listen_port: 3100

    distributor:
      ring:
        kvstore:
          store: memberlist

    memberlist:
      join_members:
        - {{ include "loki.fullname" . }}-memberlist

    ingester:
      lifecycler:
        ring:
          kvstore:
            store: memberlist
          replication_factor: 1
      chunk_target_size: 1536000
      max_chunk_age: 4h
      chunk_idle_period: 10m
      chunk_block_size: 262144
      chunk_encoding: snappy
      chunk_retain_period: 5m
      max_transfer_retries: 0

    limits_config:
      enforce_metric_name: false
      reject_old_samples: false
      reject_old_samples_max_age: 730h
      retention_period: 48h

    schema_config:
      configs:
        - from: "2020-09-07"
          store: boltdb-shipper
          object_store: filesystem
          schema: v11
          index:
            prefix: loki_index_
            period: 24h

    storage_config:
      boltdb_shipper:
        shared_store: filesystem
        active_index_directory: /var/loki/index
        cache_location: /var/loki/cache
        cache_ttl: 168h
        {{- if .Values.indexGateway.enabled }}
        index_gateway_client:
          server_address: dns:///{{ include "loki.indexGatewayFullname" . }}:9095
        {{- end }}
      filesystem:
        directory: /var/loki/chunks

    chunk_store_config:
      max_look_back_period: 10s

    table_manager:
      retention_deletes_enabled: true
      retention_period: 336h

    querier:
      query_timeout: 10m

    query_range:
      align_queries_with_step: true
      max_retries: 10
      split_queries_by_interval: 10m
      cache_results: true
      results_cache:
        cache:
          enable_fifocache: true
          fifocache:
            max_size_items: 10240
            validity: 24h

    frontend_worker:
      frontend_address: {{ include "loki.queryFrontendFullname" . }}:9095

    frontend:
      log_queries_longer_than: 5s
      compress_responses: true
      tail_proxy_url: http://{{ include "loki.querierFullname" . }}:3100

    compactor:
      working_directory: /var/loki/compactor
      shared_store: filesystem
      compaction_interval: 15m
    #      retention_enabled: true
    #      retention_delete_delay: 24h
    #      retention_delete_worker_count: 300
    ruler:
      storage:
        type: local
        local:
          directory: /etc/loki/rules
      ring:
        kvstore:
          store: memberlist
      rule_path: /tmp/loki/scratch
      alertmanager_url: http://prometheus-alertmanager.monitoring.svc.cluster.local
      enable_api: true

Using loki-distributed helm charts.

loki image : docker.io/grafana/loki:2.3.0

Did I miss something on the helm charts?

[DaveOHenry]

Ran into the same issue. Looks like a missing volume mapping / mount point. This can be fixed with the following configuration for example:

  tableManager:
    enabled: true
    replicas: 1
    extraVolumes:
    - name: bolt-db
      emptyDir: {}
    extraVolumeMounts:
    - name: bolt-db
      mountPath: /var/loki

[ebini]

hi, got the same problem. as soon as i have in storage config:

      filesystem:
        directory: /var/loki/chunks

i get this error.

[js8080]

I just ran into this same issue deploying the loki-distributed chart with local / boltdb-shipper storage.

In my case the querier pod was failing with this same error:

msg="error running loki" err="mkdir /var/loki: read-only file system

Thanks to @DaveOHenry I was able to add the extra volume mounts to my values overrides:

  querier:
    replicas: 1
    persistence:
      # -- Enable creating PVCs for the querier cache
      enabled: true
      # -- Size of persistent disk
      size: 10Gi
      # -- Storage class to be used.
      # If defined, storageClassName: <storageClass>.
      # If set to "-", storageClassName: "", which disables dynamic provisioning.
      # If empty or set to null, no storageClassName spec is
      # set, choosing the default provisioner (gp2 on AWS, standard on GKE, AWS, and OpenStack).
      storageClass: null
    extraVolumes:
    - name: bolt-db
      emptyDir: {}
    extraVolumeMounts:
    - name: bolt-db
      mountPath: /var/loki

[aslafy-z]

This was fixed by https://github.com/grafana/helm-charts/pull/1347. This issue can now be closed.

[patsevanton]

Issue not fixed.

[patsevanton]

@qdupuy Try latest version helm chart

[AurimasNav]

experiencing the same with 5.8.0 grafana/loki chart