search

grafana / tempo

Grafana Tempo is a high volume, minimal dependency distributed tracing backend.
https://grafana.com/oss/tempo/
GNU Affero General Public License v3.0
4.03k stars 522 forks source link

Getting error with tempo deployment #4339

Open Aniketsingh1 opened 1 day ago

Aniketsingh1 commented 1 day ago

The following successfully deployed tempo as daemonset on one evening: apiVersion: apps/v1 kind: DaemonSet metadata: name: tempo labels: app: tempo spec: selector: matchLabels: app: tempo template: metadata: labels: app: tempo spec: nodeSelector: nodepool: aksuser containers:

apiVersion: v1 kind: ConfigMap metadata: name: tempo-config labels: app: tempo data: tempo.yaml: | server: http_listen_port: 3200

distributor:
  receivers:
    jaeger:
      protocols:
        grpc: {}
        thrift_binary: {}
        thrift_compact: {}
        thrift_http: {}
    otlp:
      protocols:
        grpc: 
        http: {}   # OTLP HTTP receiver

ingester:
  lifecycler:
    ring:
      kvstore:
        store: memberlist
      replication_factor: 1

storage:
  trace:
    backend: local
    local:
      path: /var/tempo/traces  # Ensure storage path is consistent
    block:
      bloom_filter_false_positive: .05
    wal:
      path: /var/tempo/wal
    pool:
      max_workers: 100
      queue_depth: 10000 

compactor:
  compaction:
    compaction_window: 1h
    max_block_bytes: 100_000_000
    block_retention: 1h
    compacted_block_retention: 10m

querier:
  frontend_worker:
    frontend_address: tempo.monitoring.svc.cluster.local:9095

frontend:
  queue:
    size: 10000
    workers: 10

metrics_generator:
  processor:
    span_metrics:
      dimensions:
        - http.method
        - http.status_code
        - service.version
    service_graphs:
      dimensions:
        - http.method
        - http.status_code
        - service.version
  storage:
    path: /tmp/tempo/generator/metrics
    remote_write:
      - url: http://mimir:9009/api/v1/push
        send_exemplars: true

overrides:
  metrics_generator_processors: ['service-graphs', 'span-metrics', 'local-blocks']

apiVersion: v1 kind: PersistentVolumeClaim metadata: name: tempo-pvc labels: app: tempo spec: accessModes:

apiVersion: v1 kind: Service metadata: name: tempo labels: app: tempo spec: ports:

but suddenly i have started getting error Found 8 pods, using pod/tempo-d6mm5 failed parsing config: failed to parse configFile /etc/tempo/tempo.yaml: yaml: unmarshal errors: line 48: field frontend not found in type app.Config which i am not able to get. If i remove the section frontend: queue: size: 10000 workers: 10 then nothing appears on grafana. Please help in solving this

mapno commented 1 day ago

Hi @Aniketsingh1. The key for the frontend's config block is query_frontend, not frontend. You can refer to the config docs for more info: https://grafana.com/docs/tempo/latest/configuration/.

Aniketsingh1 commented 1 day ago

I have made the changes as per your suggestion . Please see below -

apiVersion: apps/v1 kind: DaemonSet metadata: name: tempo labels: app: tempo spec: selector: matchLabels: app: tempo template: metadata: labels: app: tempo spec: nodeSelector: nodepool: aksuser containers:

apiVersion: v1 kind: ConfigMap metadata: name: tempo-config labels: app: tempo data: tempo.yaml: | server: http_listen_port: 3200

distributor:
  receivers:
    jaeger:
      protocols:
        grpc: {}
        thrift_binary: {}
        thrift_compact: {}
        thrift_http: {}
    otlp:
      protocols:
        grpc: 
        http: {}   # OTLP HTTP receiver

ingester:
  lifecycler:
    ring:
      kvstore:
        store: memberlist
      replication_factor: 1

storage:
  trace:
    backend: local
    local:
      path: /var/tempo/traces  # Ensure storage path is consistent

compactor:
  compaction:
    compaction_window: 1h
    max_block_bytes: 100_000_000
    block_retention: 1h
    compacted_block_retention: 10m

querier:
  frontend_worker:
    frontend_address: tempo.monitoring.svc.cluster.local:9095

query_frontend:
    max_retries: 3
    max_batch_size: 10
    multi_tenant_queries_enabled: true
    search:
        concurrent_jobs: 500
        target_bytes_per_job: 104857600
        default_result_limit: 50
        max_result_limit: 1000
    metrics:
        concurrent_jobs: 500

metrics_generator:
  processor:
    span_metrics:
      dimensions:
        - http.method
        - http.status_code
        - service.version
    service_graphs:
      dimensions:
        - http.method
        - http.status_code
        - service.version
  storage:
    path: /tmp/tempo/generator/metrics
    remote_write:
      - url: http://mimir:9009/api/v1/push
        send_exemplars: true

overrides:
  metrics_generator_processors: ['service-graphs', 'span-metrics', 'local-blocks']

apiVersion: v1 kind: PersistentVolumeClaim metadata: name: tempo-pvc labels: app: tempo spec: accessModes:

apiVersion: v1 kind: Service metadata: name: tempo labels: app: tempo spec: ports:

But for this my tempo pods keep on restarting and give following logs:

Image

level=error ts=2024-11-18T11:19:35.927530275Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:35.962757781Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:35.984148428Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:36.036197128Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:36.067920294Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:36.135006167Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:36.142163249Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:36.241419494Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:36.242481506Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:36.277725112Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:36.282868671Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:36.307168951Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:36.321308214Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:36.346485605Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:36.365782627Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:36.391990929Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:36.406182093Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:36.638882275Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:36.737145508Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:36.760388976Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:36.805709398Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:36.818890547Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:36.88404808Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:36.964289283Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:36.974415997Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:36.986648234Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:37.127188615Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:37.128290227Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:37.144462909Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:37.159807982Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:37.161869705Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:37.199088624Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:37.276237891Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:37.279418227Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:37.284651586Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:37.323869227Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:37.358041011Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:37.547357641Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:37.583641349Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:37.680879242Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:37.728063973Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:37.744240555Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:37.859507451Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:37.866734833Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:37.896936372Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:37.960163883Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:37.967344064Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:37.969509889Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:37.974671847Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:38.03808636Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:38.057272176Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:38.109454362Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:38.140725814Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:38.179933555Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:38.181083368Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:38.187267838Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:38.220053006Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:38.412246868Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:38.419379148Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:38.555755882Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:38.646964708Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:38.699164695Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:38.768408674Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:38.778561788Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:38.833817509Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:38.845072036Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:38.849270383Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:38.858508087Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:38.881759449Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:38.905957521Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:38.946136973Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:38.948290897Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:38.994501617Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:39.072793797Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:39.075958833Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:39.081144791Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:39.092390418Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:39.225764418Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:39.296967619Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:39.54340049Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:39.604614479Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:39.61987035Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:39.626016919Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:39.666223372Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:39.67144853Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:39.699631847Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:39.709843162Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:39.756049382Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:39.772182563Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:39.809418182Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:39.862632181Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:39.86880585Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:39.876955942Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:39.903263538Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:39.97549115Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:40.004721879Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:40.052137812Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:40.057438372Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:40.132790519Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:40.38623777Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:40.474408261Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:40.486642499Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:40.545844165Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:40.609018775Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:40.6111902Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:40.611296401Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:40.616475759Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:40.639753621Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:40.648968125Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:40.706229269Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:40.784445748Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:40.79173993Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=error ts=2024-11-18T11:19:40.793943855Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused\"" level=warn ts=2024-11-18T11:19:44.623167023Z caller=server.go:2220 msg="GET /metrics 1.150312ms, error: write tcp 10.0.3.245:3200->10.0.2.68:36958: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:19:44.623400925Z caller=server.go:2220 msg="GET /metrics 1.392316ms, error: write tcp 10.0.3.245:3200->10.0.2.68:36960: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:19:54.623574327Z caller=server.go:2220 msg="GET /metrics 1.269814ms, error: write tcp 10.0.3.245:3200->10.0.2.68:57610: write: connection reset by peer ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:19:54.623644428Z caller=server.go:2220 msg="GET /metrics 1.397615ms, error: write tcp 10.0.3.245:3200->10.0.2.68:57608: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:20:04.623780857Z caller=server.go:2220 msg="GET /metrics 1.246113ms, error: write tcp 10.0.3.245:3200->10.0.2.68:56498: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:20:04.623920559Z caller=server.go:2220 msg="GET /metrics 1.385915ms, error: write tcp 10.0.3.245:3200->10.0.2.68:56500: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=info ts=2024-11-18T11:20:05.582257386Z caller=flush.go:249 msg="flushing block" tenant=single-tenant block=01898622-f55e-41f4-854a-7dc48e0ddd19 level=info ts=2024-11-18T11:20:05.582300786Z caller=flush.go:259 msg="completing block" tenant=single-tenant blockID=01898622-f55e-41f4-854a-7dc48e0ddd19 level=info ts=2024-11-18T11:20:05.590374574Z caller=flush.go:266 msg="block completed" tenant=single-tenant blockID=01898622-f55e-41f4-854a-7dc48e0ddd19 duration=8.071088ms level=error ts=2024-11-18T11:20:05.590422175Z caller=flush.go:232 org_id=single-tenant msg="error performing op in flushQueue" op=0 block=01898622-f55e-41f4-854a-7dc48e0ddd19 attempts=2 err="error completing wal block with local backend: error creating iterator for /var/tempo/wal/01898622-f55e-41f4-854a-7dc48e0ddd19+single-tenant+vParquet4/0000000001: error opening file: open /var/tempo/wal/01898622-f55e-41f4-854a-7dc48e0ddd19+single-tenant+vParquet4/0000000001: no such file or directory" level=info ts=2024-11-18T11:20:05.590442375Z caller=flush.go:394 org_id=single-tenant msg="retrying op in flushQueue" op=0 block=01898622-f55e-41f4-854a-7dc48e0ddd19 backoff=1m0s level=warn ts=2024-11-18T11:20:14.622769387Z caller=server.go:2220 msg="GET /metrics 1.070512ms, error: write tcp 10.0.3.245:3200->10.0.2.68:43506: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:20:14.625128212Z caller=server.go:2220 msg="GET /metrics 3.418737ms, error: write tcp 10.0.3.245:3200->10.0.2.68:43504: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:20:24.623634231Z caller=server.go:2220 msg="GET /metrics 1.853118ms, error: write tcp 10.0.3.245:3200->10.0.2.68:57114: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:20:24.623655431Z caller=server.go:2220 msg="GET /metrics 1.913118ms, error: write tcp 10.0.3.245:3200->10.0.2.68:57106: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=info ts=2024-11-18T11:20:29.254574454Z caller=frontend.go:333 msg="received shutdown notification from querier" querier=tempo-2qjzf level=warn ts=2024-11-18T11:20:34.623334276Z caller=server.go:2220 msg="GET /metrics 1.150412ms, error: write tcp 10.0.3.245:3200->10.0.2.68:48226: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:20:34.623672879Z caller=server.go:2220 msg="GET /metrics 1.490315ms, error: write tcp 10.0.3.245:3200->10.0.2.68:48228: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:20:44.62358927Z caller=server.go:2220 msg="GET /metrics 1.335614ms, error: write tcp 10.0.3.245:3200->10.0.2.68:49110: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:20:44.623890673Z caller=server.go:2220 msg="GET /metrics 1.638117ms, error: write tcp 10.0.3.245:3200->10.0.2.68:49112: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:20:54.623031183Z caller=server.go:2220 msg="GET /metrics 1.105712ms, error: write tcp 10.0.3.245:3200->10.0.2.68:46412: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:20:54.623058383Z caller=server.go:2220 msg="GET /metrics 1.144012ms, error: write tcp 10.0.3.245:3200->10.0.2.68:46408: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:21:04.623630028Z caller=server.go:2220 msg="GET /metrics 1.419915ms, error: write tcp 10.0.3.245:3200->10.0.2.68:40514: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:21:04.623631128Z caller=server.go:2220 msg="GET /metrics 1.419515ms, error: write tcp 10.0.3.245:3200->10.0.2.68:40512: write: connection reset by peer ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=info ts=2024-11-18T11:21:05.591470803Z caller=flush.go:249 msg="flushing block" tenant=single-tenant block=01898622-f55e-41f4-854a-7dc48e0ddd19 level=info ts=2024-11-18T11:21:05.591511803Z caller=flush.go:259 msg="completing block" tenant=single-tenant blockID=01898622-f55e-41f4-854a-7dc48e0ddd19 level=info ts=2024-11-18T11:21:05.597519863Z caller=flush.go:266 msg="block completed" tenant=single-tenant blockID=01898622-f55e-41f4-854a-7dc48e0ddd19 duration=6.00636ms level=error ts=2024-11-18T11:21:05.597558063Z caller=flush.go:232 org_id=single-tenant msg="error performing op in flushQueue" op=0 block=01898622-f55e-41f4-854a-7dc48e0ddd19 attempts=3 err="error completing wal block with local backend: error creating iterator for /var/tempo/wal/01898622-f55e-41f4-854a-7dc48e0ddd19+single-tenant+vParquet4/0000000001: error opening file: open /var/tempo/wal/01898622-f55e-41f4-854a-7dc48e0ddd19+single-tenant+vParquet4/0000000001: no such file or directory" level=error ts=2024-11-18T11:21:05.597577163Z caller=flush.go:271 org_id=single-tenant msg="Block exceeded max completion errors. Deleting. POSSIBLE DATA LOSS" tenant=single-tenant attempts=3 block=01898622-f55e-41f4-854a-7dc48e0ddd19 level=warn ts=2024-11-18T11:21:14.622929646Z caller=server.go:2220 msg="GET /metrics 1.202813ms, error: write tcp 10.0.3.245:3200->10.0.2.68:46666: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:21:14.623037447Z caller=server.go:2220 msg="GET /metrics 1.310414ms, error: write tcp 10.0.3.245:3200->10.0.2.68:46660: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:21:24.623203111Z caller=server.go:2220 msg="GET /metrics 1.272313ms, error: write tcp 10.0.3.245:3200->10.0.2.68:34660: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:21:24.623257111Z caller=server.go:2220 msg="GET /metrics 1.325514ms, error: write tcp 10.0.3.245:3200->10.0.2.68:34670: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:21:34.62358787Z caller=server.go:2220 msg="GET /metrics 1.104211ms, error: write tcp 10.0.3.245:3200->10.0.2.68:50592: write: connection reset by peer ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:21:34.623730871Z caller=server.go:2220 msg="GET /metrics 1.255412ms, error: write tcp 10.0.3.245:3200->10.0.2.68:50584: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=info ts=2024-11-18T11:21:44.281346334Z caller=frontend.go:333 msg="received shutdown notification from querier" querier=tempo-2qjzf level=warn ts=2024-11-18T11:21:44.62361411Z caller=server.go:2220 msg="GET /metrics 1.287814ms, error: write tcp 10.0.3.245:3200->10.0.2.68:46508: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:21:44.623793612Z caller=server.go:2220 msg="GET /metrics 1.445616ms, error: write tcp 10.0.3.245:3200->10.0.2.68:46510: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:21:54.623922186Z caller=server.go:2220 msg="GET /metrics 1.404715ms, error: write tcp 10.0.3.245:3200->10.0.2.68:58170: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:21:54.624261789Z caller=server.go:2220 msg="GET /metrics 1.686417ms, error: write tcp 10.0.3.245:3200->10.0.2.68:58168: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:22:04.623180231Z caller=server.go:2220 msg="GET /metrics 1.292513ms, error: write tcp 10.0.3.245:3200->10.0.2.68:39456: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:22:04.623201131Z caller=server.go:2220 msg="GET /metrics 1.328614ms, error: write tcp 10.0.3.245:3200->10.0.2.68:39454: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:22:14.623128284Z caller=server.go:2220 msg="GET /metrics 1.221013ms, error: write tcp 10.0.3.245:3200->10.0.2.68:36272: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:22:14.623128284Z caller=server.go:2220 msg="GET /metrics 1.224213ms, error: write tcp 10.0.3.245:3200->10.0.2.68:36274: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:22:24.623292779Z caller=server.go:2220 msg="GET /metrics 1.205912ms, error: write tcp 10.0.3.245:3200->10.0.2.68:57274: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:22:24.623646183Z caller=server.go:2220 msg="GET /metrics 1.547716ms, error: write tcp 10.0.3.245:3200->10.0.2.68:57262: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:22:34.623536838Z caller=server.go:2220 msg="GET /metrics 1.107511ms, error: write tcp 10.0.3.245:3200->10.0.2.68:39218: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:22:34.623778641Z caller=server.go:2220 msg="GET /metrics 1.413914ms, error: write tcp 10.0.3.245:3200->10.0.2.68:39228: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:22:44.62401825Z caller=server.go:2220 msg="GET /metrics 1.275614ms, error: write tcp 10.0.3.245:3200->10.0.2.68:60368: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:22:44.62403795Z caller=server.go:2220 msg="GET /metrics 1.243013ms, error: write tcp 10.0.3.245:3200->10.0.2.68:60362: write: connection reset by peer ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:22:54.623255613Z caller=server.go:2220 msg="GET /metrics 1.140011ms, error: write tcp 10.0.3.245:3200->10.0.2.68:57232: write: connection reset by peer ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:22:54.623346214Z caller=server.go:2220 msg="GET /metrics 1.231112ms, error: write tcp 10.0.3.245:3200->10.0.2.68:57234: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:23:04.623114383Z caller=server.go:2220 msg="GET /metrics 1.341114ms, error: write tcp 10.0.3.245:3200->10.0.2.68:54498: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:23:04.623107383Z caller=server.go:2220 msg="GET /metrics 1.350614ms, error: write tcp 10.0.3.245:3200->10.0.2.68:54500: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:23:14.623407884Z caller=server.go:2220 msg="GET /metrics 1.036011ms, error: write tcp 10.0.3.245:3200->10.0.2.68:49800: write: connection reset by peer ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:23:14.623735987Z caller=server.go:2220 msg="GET /metrics 1.363314ms, error: write tcp 10.0.3.245:3200->10.0.2.68:49790: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=info ts=2024-11-18T11:23:18.230597174Z caller=frontend.go:333 msg="received shutdown notification from querier" querier=tempo-2qjzf level=warn ts=2024-11-18T11:23:24.623086562Z caller=server.go:2220 msg="GET /metrics 1.128811ms, error: write tcp 10.0.3.245:3200->10.0.2.68:46958: write: connection reset by peer ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; " level=warn ts=2024-11-18T11:23:24.623173563Z caller=server.go:2220 msg="GET /metrics 1.219712ms, error: write tcp 10.0.3.245:3200->10.0.2.68:46956: write: connection reset by peer ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "

Am i missing something?