loki error caller=flush.go:220 org_id=fake msg="failed to flush user"

[Liang-Zai]

level=error ts=2024-06-18T05:31:21.309015445Z caller=flush.go:220 org_id=fake msg="failed to flush user" err="open /data/loki/chunks/ZmFrZS9hYmQzZTYwMDBjOWI0OWQyOjE5MDI5Y2QzMjk2OjE5MDI5Y2Q0NzA3OmI3ZWMxYjE3: no space left on device" level=error ts=2024-06-18T05:31:21.309073172Z caller=flush.go:220 org_id=fake msg="failed to flush user" err="open /data/loki/chunks/ZmFrZS9lOGJjOWI0YTllNWI0YjhmOjE5MDI3OWUxN2RmOjE5MDI3OWZmODkwOjMxZjVhZTA=: no space left on device" level=error ts=2024-06-18T05:31:21.309151159Z caller=flush.go:220 org_id=fake msg="failed to flush user" err="open /data/loki/chunks/ZmFrZS8xZTNhNzVhN2E0MDNkMGNkOjE5MDI4YjJjMDBmOjE5MDI4YjNhODkzOmIyMjBlMDZm: no space left on device" level=error ts=2024-06-18T05:31:21.309226812Z caller=flush.go:220 org_id=fake msg="failed to flush user" err="open /data/loki/chunks/ZmFrZS8zY2Q4ZDEyODA3YjQyNWU2OjE5MDI5NjNlNTIwOjE5MDI5NjYyZTdiOmNhNTNkZGU=: no space left on device" level=error ts=2024-06-18T05:31:21.309291007Z caller=flush.go:220 org_id=fake msg="failed to flush user" err="open /data/loki/chunks/ZmFrZS83OTAzYjE3MjJjZTg2MWY0OjE5MDI4MmJmZWQ2OjE5MDI4MmMwMDAwOmQzMTVkMWRi: no space left on device" level=error ts=2024-06-18T05:31:21.309348267Z caller=flush.go:220 org_id=fake msg="failed to flush user" err="open /data/loki/chunks/ZmFrZS9kNjI0MGVhMTA2MDhhOTM5OjE5MDI3YTFkNTM2OjE5MDI3YTFkN2JmOjY0MDZhZjU0: no space left on device" level=error ts=2024-06-18T05:31:21.309415316Z caller=flush.go:220 org_id=fake msg="failed to flush user" err="open /data/loki/chunks/ZmFrZS8zZjg2Y2YwNGRkNjEzY2Y5OjE5MDI5N2UxNDViOjE5MDI5N2ZlOTM3OjY1YTI4N2Ey: no space left on device" level=error ts=2024-06-18T05:31:21.309485367Z caller=flush.go:220 org_id=fake msg="failed to flush user" err="open /data/loki/chunks/ZmFrZS81YTM5YjY2Zjc4YzQ3NzRkOjE5MDI5MjUxMGFhOjE5MDI5MjU4MDA3OmE0OWJjZDA4: no space left on device" level=error ts=2024-06-18T05:31:21.309552694Z caller=flush.go:220 org_id=fake msg="failed to flush user" err="open /data/loki/chunks/ZmFrZS81MDFhM2M4NzEzOGI4MmFhOjE5MDI2ZGY3MzEwOjE5MDI2ZTFhMWZhOjgxYTJkYTFj: no space left on device" level=error ts=2024-06-18T05:31:21.309638848Z caller=flush.go:220 org_id=fake msg="failed to flush user" err="open /data/loki/chunks/ZmFrZS9kMTZmMDhlZTgwNjQ3MDI3OjE5MDI4NzU0MzZlOjE5MDI4NzcxODVlOjY5NThhMDNj: no space left on device"

How to handle the error above

[jtackaberry]

Add more space to the volume where /data/loki/chunks is located?

[Liang-Zai]

Filesystem Size Used Avail Use% Mounted on /dev/sdb1 30T 13T 16T 44% /data

Storage space should still be sufficient

[jtackaberry]

This is almost certainly environmental, not related to Loki.

Exec into the pod and check df -h /data/loki from within the pod. Try to touch a file under that path as well.

[Liang-Zai]

When I touch test.txt, it works fine

[Liang-Zai]

loki-statefulset.yaml ''' apiVersion: v1 kind: Service metadata: name: loki namespace: dhome-monitor labels: app: loki spec: type: NodePort ports:

• port: 3100 protocol: TCP name: http-metrics targetPort: http-metrics nodePort: 30100 selector: app: loki

  apiVersion: apps/v1 kind: StatefulSet metadata: name: loki namespace: dhome-monitor labels: app: loki spec: podManagementPolicy: OrderedReady replicas: 1 selector: matchLabels: app: loki serviceName: loki updateStrategy: type: RollingUpdate template: metadata: labels: app: loki spec: serviceAccountName: loki securityContext: fsGroup: 10001 runAsGroup: 10001 runAsNonRoot: true runAsUser: 10001 imagePullSecrets:

  • name: uto-jfrog nodeName: appuser containers:
  • name: loki image: jfrog.utopilot.com.cn/docker/platform/loki:2.3.0 imagePullPolicy: IfNotPresent args:
    • -config.file=/etc/loki/loki.yaml resources: limits: cpu: '6' memory: "40Gi" requests: cpu: '2' memory: "4Gi" volumeMounts:
    • name: config mountPath: /etc/loki
    • name: storage mountPath: /data ports:
    • name: http-metrics containerPort: 3100 protocol: TCP livenessProbe: httpGet: path: /ready port: http-metrics scheme: HTTP initialDelaySeconds: 45 readinessProbe: httpGet: path: /ready port: http-metrics scheme: HTTP initialDelaySeconds: 45 securityContext: readOnlyRootFilesystem: true terminationGracePeriodSeconds: 4800 volumes:
  • name: config configMap: name: loki
  • name: storage hostPath: path: /data/dhonemonitor/loki '''

loki-rbac.yaml ''' apiVersion: v1 kind: ServiceAccount metadata: name: loki namespace: dhome-monitor

apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: loki namespace: dhome-monitor rules:

• apiGroups: [ "extensions" ] resources: [ "podsecuritypolicies" ] verbs: [ "use" ] resourceNames: [ loki ]

  apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: loki namespace: dhome-monitor roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: loki subjects:

• kind: ServiceAccount name: loki '''

loki-configmap.yaml ''' apiVersion: v1 kind: ConfigMap metadata: name: loki namespace: dhome-monitor labels: app: loki data: loki.yaml: | auth_enabled: false ingester: chunk_idle_period: 2m chunk_block_size: 262144 chunk_retain_period: 0s max_transfer_retries: 0 chunk_target_size: 1572864
max_chunk_age: 2m chunk_encoding: snappy concurrent_flushes: 128 flush_check_period: 10s flush_op_timeout: 2m lifecycler: ring: kvstore: store: memberlist
replication_factor: 1
limits_config: enforce_metric_name: false reject_old_samples: true reject_old_samples_max_age: 168h
max_streams_matchers_per_query: 100000 max_entries_limit_per_query: 0 schema_config: configs:

• from: "2022-05-01" store: boltdb-shipper
  objectstore: filesystem schema: v11 index: prefix: index
  period: 24h
  chunks: period: 24h server: http_listen_port: 3100 storage_config: boltdb_shipper: active_index_directory: /data/loki/boltdb-shipper-active cache_location: /data/loki/boltdb-shipper-cache cache_ttl: 24h shared_store: filesystem filesystem: directory: /data/loki/chunks index_cache_validity: 1m max_chunk_batch_size: 60 chunk_store_config: max_look_back_period: 0s
  table_manager: retention_deletes_enabled: true
  retention_period: 4320h
  compactor: working_directory: /data/loki/boltdb-shipper-compactor shared_store: filesystem

'''

This is my loki configuration, I wonder if it is a configuration problem