Query Frontend Error Messages: caller=frontend_processor.go:145 msg="error processing requests" err=EOF

We are running a bare-metal instance of Loki with separated read/write paths. The queriers use a DNS name for the query frontend that resolves to the querier IPs.

When starting the Loki service the querier connects to the query frontend and states that it's ready but error messages are repeatedly popping up that indicate it's not working properly.

Loki v2.9.4 loki.yml: --- start config --- target: read auth_enabled: true common: replication_factor: 3 ring: heartbeat_timeout: 10m kvstore: store: memberlist storage: s3: access_key_id: omitted bucketnames: loki endpoint: omitted http_config: insecure_skip_verify: true insecure: false region: default s3forcepathstyle: true secret_access_key: omitted compactor: compaction_interval: 1m retention_enabled: true shared_store: s3 working_directory: /loki/compactor frontend: compress_responses: true log_queries_longer_than: 15s frontend_worker: frontend_address: frontend.loki.it.ufl.edu:9443 grpc_client_config: max_send_msg_size: 104857600.0 parallelism: 12 ingester: chunk_idle_period: 1h flush_check_period: 10s max_chunk_age: 2h wal: replay_memory_ceiling: 24042MB limits_config: enforce_metric_name: false ingestion_burst_size_mb: 60 ingestion_rate_mb: 40 max_cache_freshness_per_query: 10m max_entries_limit_per_query: 100000 max_global_streams_per_user: 20000 max_query_parallelism: 32 max_query_series: 10000 per_stream_rate_limit: 40MB per_stream_rate_limit_burst: 60MB query_timeout: 3m reject_old_samples: true retention_period: 2w split_queries_by_interval: 15m memberlist: abort_if_cluster_join_fails: false bind_port: 7946 join_members:

az1-irs-o11y-prod-loki-write-01.server.ufl.edu
az1-irs-o11y-prod-loki-write-02.server.ufl.edu
az1-irs-o11y-prod-loki-write-03.server.ufl.edu
az1-irs-o11y-prod-loki-write-04.server.ufl.edu
az2-irs-o11y-prod-loki-write-01.server.ufl.edu
az2-irs-o11y-prod-loki-write-02.server.ufl.edu
az2-irs-o11y-prod-loki-write-03.server.ufl.edu
az2-irs-o11y-prod-loki-write-04.server.ufl.edu
az1-irs-o11y-prod-loki-read-01.server.ufl.edu
az1-irs-o11y-prod-loki-read-02.server.ufl.edu
az1-irs-o11y-prod-loki-read-03.server.ufl.edu
az2-irs-o11y-prod-loki-read-01.server.ufl.edu
az2-irs-o11y-prod-loki-read-02.server.ufl.edu
az2-irs-o11y-prod-loki-read-03.server.ufl.edu max_join_backoff: 1m max_join_retries: 10 min_join_backoff: 1s rejoin_interval: 1m querier: max_concurrent: 6000 multi_tenant_queries_enabled: true query_range: align_queries_with_step: true cache_results: true max_retries: 5 results_cache: cache: embedded_cache: enabled: true max_size_mb: 2048 ttl: 1h query_scheduler: max_outstanding_requests_per_tenant: 42768 schema_config: configs:
from: '2023-03-15' index: period: 24h prefix: indextsdb object_store: s3 schema: v12 store: tsdb server: grpc_listen_port: 9443 grpc_server_max_concurrent_streams: 1500 grpc_server_max_recv_msg_size: 104857600 grpc_server_max_send_msg_size: 104857600 http_listen_port: 8443 http_server_read_timeout: 610s http_server_write_timeout: 610s http_tls_config: cert_file: /etc/loki/ssl/cert.crt key_file: /etc/loki/ssl/cert.key log_level: info storage_config: hedging: at: 250ms max_per_second: 20 up_to: 3 tsdb_shipper: active_index_directory: /loki/tsdb-shipper-active cache_location: /loki/tsdb-shipper-cache shared_store: s3 --- end config ---

The read nodes are behind an nginx reverse proxy for load-balancing.

Error messages:

error: code = Canceled desc = context canceled" Mar 21 18:03:02 az1-irs-o11y-prod-loki-read-01 loki[2280155]: level=error ts=2024-03-21T18:03:02.199245853Z caller=frontend_processor.go:69 msg="error processing requests" address=10.51.31.42:9443 err="rpc error: code = Canceled desc = context canceled" Mar 21 18:03:02 az1-irs-o11y-prod-loki-read-01 loki[2280155]: level=error ts=2024-03-21T18:03:02.199267163Z caller=frontend_processor.go:69 msg="error processing requests" address=10.51.31.42:9443 err="rpc error: code = Canceled desc = context canceled" Mar 21 18:03:02 az1-irs-o11y-prod-loki-read-01 loki[2280155]: level=error ts=2024-03-21T18:03:02.895233268Z caller=frontend_processor.go:145 msg="error processing requests" err=EOF Mar 21 18:03:03 az1-irs-o11y-prod-loki-read-01 loki[2280155]: level=error ts=2024-03-21T18:03:03.253163462Z caller=frontend_processor.go:69 msg="error processing requests" address=10.51.156.34:9443 err="rpc error: code = Canceled desc = context canceled" Mar 21 18:03:03 az1-irs-o11y-prod-loki-read-01 loki[2280155]: level=error ts=2024-03-21T18:03:03.253198493Z caller=frontend_processor.go:69 msg="error processing requests" address=10.51.156.34:9443 err="rpc error: code = Canceled desc = context canceled" Mar 21 18:03:03 az1-irs-o11y-prod-loki-read-01 loki[2280155]: level=error ts=2024-03-21T18:03:03.556305536Z caller=frontend_processor.go:145 msg="error processing requests" err=EOF Mar 21 18:03:03 az1-irs-o11y-prod-loki-read-01 loki[2280155]: level=error ts=2024-03-21T18:03:03.607206314Z caller=frontend_processor.go:145 msg="error processing requests" err=EOF Mar 21 18:03:03 az1-irs-o11y-prod-loki-read-01 loki[2280155]: level=error ts=2024-03-21T18:03:03.786873074Z caller=frontend_processor.go:145 msg="error processing requests" err=EOF Mar 21 18:03:03 az1-irs-o11y-prod-loki-read-01 loki[2280155]: level=error ts=2024-03-21T18:03:03.790565744Z caller=frontend_processor.go:145 msg="error processing requests" err=EOF

Let me know if there are any other logs I should be looking at to determine what is going on in my environment.

Thanks!

grafana / loki

Query Frontend Error Messages: caller=frontend_processor.go:145 msg="error processing requests" err=EOF #12299