Error in to ingest logs into ingester

[kunalmehta-eve]

We are seeing following error in loki logs

level=error ts=2024-05-16T08:35:43.267554629Z caller=manager.go:49 component=distributor path=write msg="write operation failed" details="Max entry size '262144' bytes exceeded for stream '{app=\"generate-preview-5jp7j\", container=\"main\", filename=\"/var/log/pods/argo-workflows_generate-preview-5jp7j_31f70b67-5db3-4933-ab9b-0d4513b46316/main/0.log\", job=\"argo-workflows/generate-preview-5jp7j\", namespace=\"argo-workflows\", node_name=\"aks-defaultgreen-11165910-vmss00008s\", pod=\"generate-preview-5jp7j\", stream=\"stderr\"}' while adding an entry with length '583758' bytes" org_id=fake

To Reproduce Steps to reproduce the behavior:

1. Started Loki (3.0.0)
2. Started Promtail (2.9.2) to tail '...'

Expected behavior Please provide necessary configuration to fix this issue

Environment:

• Infrastructure: kubernetes
• Deployment tool: helm

Screenshots, Promtail config, or terminal output loki: auth_enabled: false analytics: reporting_enabled: false storage: type: azure azure: accountName: ${loki_azurerm_storage_account_name} bucketNames: chunks: ${loki_chunks_azurerm_storage_container_name} ruler: ${loki_ruler_azurerm_storage_container_name} admin: ${loki_admin_azurerm_storage_container_name} ingester: max_chunk_age: 24h

ingester_client: grpc_client_config: grpc_keepalive_time: 30s # Adjust keepalive settings grpc_keepalive_timeout: 20s # Adjust keepalive settings

structuredConfig: query_range: parallelise_shardable_queries: false server: http_server_write_timeout: 10m

limits_config: allow_structured_metadata: false max_concurrent_tail_requests: 100 discover_service_name: [ ] schemaConfig: configs:

• from: 2022-01-11 store: boltdb-shipper object_store: azure schema: v12 index: prefix: lokiindex period: 24h

lokiCanary: tolerations:

• key: "stack" operator: "Equal" value: "monitoring" effect: "NoSchedule" resources: requests: cpu: "0.01" memory: 64Mi limits: cpu: "0.05" memory: 128Mi

monitoring: enabled: true selfMonitoring: enabled: true grafanaAgent: installOperator: true tolerations:

• key: "stack" operator: "Equal" value: "monitoring" effect: "NoSchedule"

write: nodeSelector: stack: monitoring tolerations:

• key: "stack" operator: "Equal" value: "monitoring" effect: "NoSchedule" replicas: 3 resources: requests: cpu: "0.2" memory: 4Gi limits: cpu: "1" memory: 4Gi

read: nodeSelector: stack: monitoring tolerations:

• key: "stack" operator: "Equal" value: "monitoring" effect: "NoSchedule" replicas: 3 resources: requests: cpu: "0.2" memory: 3Gi limits: cpu: "3" memory: 8Gi

backend: nodeSelector: stack: monitoring tolerations:

• key: "stack" operator: "Equal" value: "monitoring" effect: "NoSchedule" replicas: 3 resources: requests: cpu: "0.1" memory: 512Mi limits: cpu: "0.2" memory: 1Gi

chunksCache: nodeSelector: stack: monitoring tolerations:

• key: "stack" operator: "Equal" value: "monitoring" effect: "NoSchedule" resources: requests: cpu: "0.1" memory: 1Gi limits: cpu: "0.5" memory: 3Gi

resultsCache: nodeSelector: stack: monitoring tolerations:

• key: "stack" operator: "Equal" value: "monitoring" effect: "NoSchedule" resources: requests: cpu: "0.2" memory: 64Mi limits: cpu: "0.5" memory: 256Mi

[kunalmehta-eve]

@slim-bean can you please suggest

[JStickler]

Please provide necessary configuration to fix this issue

Questions have a better chance of being answered if you ask them on the community forums.

[zhujintao]

limits_config:
  max_line_size: 0