search

grafana / pyroscope

Continuous Profiling Platform. Debug performance issues down to a single line of code
https://grafana.com/oss/pyroscope/
GNU Affero General Public License v3.0
9.97k stars 597 forks source link

Restart the pyroscope service and find that 40 minutes of data will be lost #3372

Open z617015032 opened 3 months ago

z617015032 commented 3 months ago

Describe the bug

Restart the pyroscope service and find that 40 minutes of data will be lost

To Reproduce

Steps to reproduce the behavior: yes

  1. Start Pyroscope (SHA or version): version 1.6.0
  2. Perform Operations(Read/Write/Others): /usr/bin/pyroscope -config.file=/etc/pyroscope/config.yaml

Expected behavior

Restart the pyroscope service and find that 40 minutes of data will be lost

image

config.yaml

server:
  http_listen_port: 4040
pyroscopedb:
  data_path: "/data"
  min_disk_available_percentage: 0.25
limits:
  max_query_length: 3d
  max_global_series_per_tenant: 10000
kolesnikovae commented 3 months ago

Hi @z617015032! Thank you for reporting the issue.

Could you please post the logs of that? I'm curious to see if the restart was graceful.

z617015032 commented 3 months ago

Stop mode: /usr/local/bin/docker-compose -f docker-compose.yml stop docker-compose.yaml

version: '3.9'
services:
  pyroscope:
    image: 'grafana/pyroscope:latest'
    deploy:
      resources:
        limits:
           cpus: "4"
           memory: 13G
        reservations:
            memory: 500M    
    ports:
      - 3100:4040
    volumes:
      - ./config.yaml:/etc/pyroscope/config.yaml
      - ./data:/data
  agent:
    image: grafana/agent:latest
    deploy:
      resources:
        limits:
           cpus: "7"
           memory: 30G
        reservations:
            memory: 500M    
    volumes:
      - ./agent/config:/etc/agent-config
    command:
      - run
      - /etc/agent-config/config.river
      - --server.http.listen-addr=0.0.0.0:12345
    environment:
      HOSTNAME: agent
      AGENT_MODE: flow
    ports:
      - "12345:12345"

error log

ts=2024-06-20T10:45:41.854729332Z caller=signals.go:62 level=info msg="=== received SIGINT/SIGTERM ===\n*** exiting"
ts=2024-06-20T10:45:41.854963065Z caller=basic_lifecycler.go:238 level=info component=scheduler msg="ring lifecycler is shutting down" ring=query-scheduler
ts=2024-06-20T10:45:41.855059912Z caller=basic_lifecycler.go:238 level=info component=distributor msg="ring lifecycler is shutting down" ring=distributor
ts=2024-06-20T10:45:41.855107687Z caller=module_service.go:120 level=info msg="module stopped" module=tenant-settings
ts=2024-06-20T10:45:41.855141151Z caller=basic_lifecycler.go:403 level=info component=scheduler msg="unregistering instance from ring" ring=query-scheduler
ts=2024-06-20T10:45:41.855187039Z caller=basic_lifecycler.go:403 level=info component=distributor msg="unregistering instance from ring" ring=distributor
ts=2024-06-20T10:45:41.855229164Z caller=basic_lifecycler.go:278 level=info component=distributor msg="instance removed from the ring" ring=distributor
ts=2024-06-20T10:45:41.855298921Z caller=basic_lifecycler.go:278 level=info component=scheduler msg="instance removed from the ring" ring=query-scheduler
ts=2024-06-20T10:45:41.855323684Z caller=module_service.go:120 level=info msg="module stopped" module=distributor
ts=2024-06-20T10:45:41.855386136Z caller=lifecycler.go:533 level=info msg="lifecycler loop() exited gracefully" ring=ingester
ts=2024-06-20T10:45:41.855414687Z caller=lifecycler.go:907 level=info msg="changing instance state from" old_state=ACTIVE new_state=LEAVING ring=ingester
ts=2024-06-20T10:45:41.855445026Z caller=module_service.go:120 level=info msg="module stopped" module=query-frontend
ts=2024-06-20T10:45:41.855455713Z caller=module_service.go:120 level=info msg="module stopped" module=query-scheduler
ts=2024-06-20T10:45:41.855474008Z caller=lifecycler.go:978 level=info msg="transfers are disabled"
ts=2024-06-20T10:45:41.855519701Z caller=basic_lifecycler.go:238 level=info component=overrides-exporter msg="ring lifecycler is shutting down" ring=overrides-exporter
ts=2024-06-20T10:45:41.855610032Z caller=basic_lifecycler.go:272 level=info component=overrides-exporter msg="keeping instance the ring" ring=overrides-exporter
ts=2024-06-20T10:45:41.855639208Z caller=module_service.go:120 level=info msg="module stopped" module=overrides-exporter
ts=2024-06-20T10:45:41.856029697Z caller=scheduler.go:479 level=info component=scheduler msg="received shutdown notification from querier" querier=197078a82dfa
ts=2024-06-20T10:45:41.856442019Z caller=module_service.go:120 level=info msg="module stopped" module=querier
ts=2024-06-20T10:45:41.856516669Z caller=module_service.go:120 level=info msg="module stopped" module=ring
ts=2024-06-20T10:45:41.856651497Z caller=http.go:261 level=warn traceID=4d93ea0277a1f912 msg="POST /push.v1.PusherService/Push (500) 10.436145ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:41.864009552Z caller=http.go:261 level=warn traceID=7e4b4fe15e91d92d msg="POST /push.v1.PusherService/Push (500) 392.275µs Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:41.890965591Z caller=http.go:261 level=warn traceID=6a5c9399c5f46590 msg="POST /push.v1.PusherService/Push (500) 1.069121ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:41.905701302Z caller=http.go:261 level=warn traceID=7d4f99c436c4aa72 msg="POST /push.v1.PusherService/Push (500) 1.954288ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:41.974917785Z caller=http.go:261 level=warn traceID=66884dbb0230558b msg="POST /push.v1.PusherService/Push (500) 7.032215ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:42.011140022Z caller=http.go:261 level=warn traceID=4a933306f4e23c20 msg="POST /push.v1.PusherService/Push (500) 28.217832ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:42.059112523Z caller=http.go:261 level=warn traceID=5a14d196e518169b msg="POST /push.v1.PusherService/Push (500) 5.651796ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:42.090454476Z caller=http.go:261 level=warn traceID=1d3e9c9336ff23f8 msg="POST /push.v1.PusherService/Push (500) 2.561264ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:42.128966788Z caller=http.go:261 level=warn traceID=43b6c255411836cb msg="POST /push.v1.PusherService/Push (500) 7.584843ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:42.177092155Z caller=http.go:261 level=warn traceID=0a48d6ce66ee41a0 msg="POST /push.v1.PusherService/Push (500) 3.027282ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:42.18290318Z caller=http.go:261 level=warn traceID=062c9c951662dd10 msg="POST /push.v1.PusherService/Push (500) 11.000716ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:42.201806753Z caller=http.go:261 level=warn traceID=7b8f1f0165cad5aa msg="POST /push.v1.PusherService/Push (500) 999.206µs Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:42.254030733Z caller=http.go:261 level=warn traceID=71c4962abac82630 msg="POST /push.v1.PusherService/Push (500) 5.942625ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:42.278678647Z caller=http.go:261 level=warn traceID=32fa8a2384873428 msg="POST /push.v1.PusherService/Push (500) 5.979663ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:42.325686164Z caller=http.go:261 level=warn traceID=447643424cad4329 msg="POST /push.v1.PusherService/Push (500) 1.152211ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:42.393912093Z caller=http.go:261 level=warn traceID=7243a6f6dd58dbd8 msg="POST /push.v1.PusherService/Push (500) 9.796105ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:42.418382925Z caller=http.go:261 level=warn traceID=47f4ab3f08b8d9e8 msg="POST /push.v1.PusherService/Push (500) 3.001166ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:42.427081147Z caller=http.go:261 level=warn traceID=155474851c1aad31 msg="POST /push.v1.PusherService/Push (500) 2.28593ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:42.440964983Z caller=http.go:261 level=warn traceID=52ede1461280f696 msg="POST /push.v1.PusherService/Push (500) 353.533µs Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:42.50467058Z caller=http.go:261 level=warn traceID=09d4ecc64841e75a msg="POST /push.v1.PusherService/Push (500) 3.491364ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:42.63154246Z caller=http.go:261 level=warn traceID=6ef4712a6da789d6 msg="POST /push.v1.PusherService/Push (500) 9.472929ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:42.668356766Z caller=http.go:261 level=warn traceID=4971aaab196deaae msg="POST /push.v1.PusherService/Push (500) 1.95084ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:42.689346193Z caller=http.go:261 level=warn traceID=5dee1d745695761b msg="POST /push.v1.PusherService/Push (500) 3.464519ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:42.722849941Z caller=http.go:261 level=warn traceID=2166444c778fc5cd msg="POST /push.v1.PusherService/Push (500) 1.679611ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:42.757752603Z caller=http.go:261 level=warn traceID=756f4aaefa6e9945 msg="POST /push.v1.PusherService/Push (500) 10.968382ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:42.780145349Z caller=http.go:261 level=warn traceID=5eb3b9e665276ea4 msg="POST /push.v1.PusherService/Push (500) 5.563735ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:42.78335323Z caller=http.go:261 level=warn traceID=37f852f6a534a7bc msg="POST /push.v1.PusherService/Push (500) 2.821564ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:42.806844558Z caller=http.go:261 level=warn traceID=002d378bac4b2d45 msg="POST /push.v1.PusherService/Push (500) 5.8266ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:42.817509864Z caller=http.go:261 level=warn traceID=684acf95d29427ef msg="POST /push.v1.PusherService/Push (500) 7.160305ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:42.826260354Z caller=http.go:261 level=warn traceID=39657abb5f07953e msg="POST /push.v1.PusherService/Push (500) 2.209957ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:42.82832959Z caller=http.go:261 level=warn traceID=7fbbf82136284b93 msg="POST /push.v1.PusherService/Push (500) 675.826µs Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:42.847729543Z caller=http.go:261 level=warn traceID=2d0d318816032d23 msg="POST /push.v1.PusherService/Push (500) 26.654777ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:42.858841194Z caller=http.go:261 level=warn traceID=2a0bdfe4d27bf7be msg="POST /push.v1.PusherService/Push (500) 3.009372ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:42.859432952Z caller=http.go:261 level=warn traceID=302467f0de8a6d6b msg="POST /push.v1.PusherService/Push (500) 6.16788ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:42.860234007Z caller=http.go:261 level=warn traceID=76d12e3b1e581626 msg="POST /push.v1.PusherService/Push (500) 1.591421ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:42.86778596Z caller=http.go:261 level=warn traceID=46536a6eebce76a6 msg="POST /push.v1.PusherService/Push (500) 2.085475ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:42.921190435Z caller=http.go:261 level=warn traceID=445024ea5e5c7c57 msg="POST /push.v1.PusherService/Push (500) 18.891102ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:42.923971272Z caller=http.go:261 level=warn traceID=61d6b7ab19399b1b msg="POST /push.v1.PusherService/Push (500) 6.905071ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:42.957157991Z caller=http.go:261 level=warn traceID=16f8f8aa8ff594a0 msg="POST /push.v1.PusherService/Push (500) 2.311679ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:42.97137693Z caller=http.go:261 level=warn traceID=37cd0453dd0da22b msg="POST /push.v1.PusherService/Push (500) 2.600596ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:43.087310824Z caller=http.go:261 level=warn traceID=7ca8efaa0bc79eea msg="POST /push.v1.PusherService/Push (500) 5.166419ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:43.096227079Z caller=http.go:261 level=warn traceID=7767932cf0023a91 msg="POST /push.v1.PusherService/Push (500) 5.387951ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:43.141413248Z caller=http.go:261 level=warn traceID=6d77d8d97a2c1102 msg="POST /push.v1.PusherService/Push (500) 1.397314ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:43.170265675Z caller=http.go:261 level=warn traceID=75fecc4a24cdb713 msg="POST /push.v1.PusherService/Push (500) 1.906822ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:43.199764201Z caller=http.go:261 level=warn traceID=2f84465f93add14c msg="POST /push.v1.PusherService/Push (500) 2.060117ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:43.307416069Z caller=http.go:261 level=warn traceID=1dc9f6b824c6b162 msg="POST /push.v1.PusherService/Push (500) 1.945915ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:43.325578316Z caller=http.go:261 level=warn traceID=7d8a90e353bdba14 msg="POST /push.v1.PusherService/Push (500) 9.619589ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:43.367859065Z caller=http.go:261 level=warn traceID=74b51b92deb52119 msg="POST /push.v1.PusherService/Push (500) 5.588646ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:43.388645712Z caller=http.go:261 level=warn traceID=30a56b3eaf6d4e23 msg="POST /push.v1.PusherService/Push (500) 2.193395ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:43.395267702Z caller=http.go:261 level=warn traceID=77224259b1fe8842 msg="POST /push.v1.PusherService/Push (500) 3.888488ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:43.425684267Z caller=http.go:261 level=warn traceID=1742930c55636787 msg="POST /push.v1.PusherService/Push (500) 1.903649ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:43.485792066Z caller=http.go:261 level=warn traceID=5b50c35341b3ae5c msg="POST /push.v1.PusherService/Push (500) 3.533607ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:43.540963825Z caller=http.go:261 level=warn traceID=465e0e31620493f4 msg="POST /push.v1.PusherService/Push (500) 11.744051ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:43.552683183Z caller=http.go:261 level=warn traceID=7b84082bbe6342f3 msg="POST /push.v1.PusherService/Push (500) 6.164944ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:43.573372516Z caller=http.go:261 level=warn traceID=0358292f4a09ddc7 msg="POST /push.v1.PusherService/Push (500) 2.04843ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:43.573945737Z caller=http.go:261 level=warn traceID=717e535751173374 msg="POST /push.v1.PusherService/Push (500) 2.039668ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:43.592871354Z caller=http.go:261 level=warn traceID=71622d26f35b4ec7 msg="POST /push.v1.PusherService/Push (500) 9.40391ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:43.597703663Z caller=http.go:261 level=warn traceID=734e7a18bf7ecf36 msg="POST /push.v1.PusherService/Push (500) 880.382µs Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:43.616393989Z caller=http.go:261 level=warn traceID=6e00edc0f1813e63 msg="POST /push.v1.PusherService/Push (500) 5.076521ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:43.626524958Z caller=http.go:261 level=warn traceID=7718e264bf6eb8bc msg="POST /push.v1.PusherService/Push (500) 4.426197ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:43.630803451Z caller=http.go:261 level=warn traceID=65bec1fcbbfea160 msg="POST /push.v1.PusherService/Push (500) 1.281167ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:43.656405894Z caller=http.go:261 level=warn traceID=7701a5e949543995 msg="POST /push.v1.PusherService/Push (500) 1.07298ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:43.736909986Z caller=http.go:261 level=warn traceID=35359375f70b31a9 msg="POST /push.v1.PusherService/Push (500) 2.039542ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:43.787863959Z caller=http.go:261 level=warn traceID=7846a5f997c6aafa msg="POST /push.v1.PusherService/Push (500) 3.252655ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:43.801959784Z caller=http.go:261 level=warn traceID=671ef2ea348c5901 msg="POST /push.v1.PusherService/Push (500) 2.008229ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:43.803449104Z caller=http.go:261 level=warn traceID=15fed76d41d68b49 msg="POST /push.v1.PusherService/Push (500) 3.564425ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:43.828192133Z caller=http.go:261 level=warn traceID=17397dcd7e85f061 msg="POST /push.v1.PusherService/Push (500) 1.55261ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:43.910762314Z caller=http.go:261 level=warn traceID=7922b5661d73dced msg="POST /push.v1.PusherService/Push (500) 3.556943ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:43.944651582Z caller=http.go:261 level=warn traceID=18a7144184fbdbea msg="POST /push.v1.PusherService/Push (500) 359.794µs Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:43.972876138Z caller=http.go:261 level=warn traceID=617e7fc1aabbbf58 msg="POST /push.v1.PusherService/Push (500) 1.799966ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:43.998754849Z caller=http.go:261 level=warn traceID=2ea468165c832d51 msg="POST /push.v1.PusherService/Push (500) 3.377701ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:43.999178392Z caller=http.go:261 level=warn traceID=77b0faa1c4ece0b7 msg="POST /push.v1.PusherService/Push (500) 683.728µs Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:44.027381195Z caller=http.go:261 level=warn traceID=298334706b7c5106 msg="POST /push.v1.PusherService/Push (500) 1.059559ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:44.067655651Z caller=http.go:261 level=warn traceID=266e5d981bb64cef msg="POST /push.v1.PusherService/Push (500) 6.716974ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:44.103562348Z caller=http.go:261 level=warn traceID=7c0bae002eb2869d msg="POST /push.v1.PusherService/Push (500) 1.945613ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:44.103884635Z caller=http.go:261 level=warn traceID=64690c6dc12a8eff msg="POST /push.v1.PusherService/Push (500) 2.793465ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:44.116240816Z caller=http.go:261 level=warn traceID=6bf8979f550803ac msg="POST /push.v1.PusherService/Push (500) 3.86109ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:44.132018615Z caller=http.go:261 level=warn traceID=716fb4110d7c41a7 msg="POST /push.v1.PusherService/Push (500) 1.272335ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:44.157826083Z caller=http.go:261 level=warn traceID=0112094b0074d68d msg="POST /push.v1.PusherService/Push (500) 2.417891ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:44.187489667Z caller=http.go:261 level=warn traceID=45bd768498b400ec msg="POST /push.v1.PusherService/Push (500) 1.124929ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:44.209752991Z caller=http.go:261 level=warn traceID=634858b1e1bc0e8e msg="POST /push.v1.PusherService/Push (500) 2.090916ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:44.217995642Z caller=http.go:261 level=warn traceID=0887751850fa9096 msg="POST /push.v1.PusherService/Push (500) 5.806952ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:44.23866917Z caller=http.go:261 level=warn traceID=292a7e4fbf52b572 msg="POST /push.v1.PusherService/Push (500) 1.85658ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:44.2422384Z caller=http.go:261 level=warn traceID=588039cdf69a8ea2 msg="POST /push.v1.PusherService/Push (500) 2.098754ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "
ts=2024-06-20T10:45:44.246484652Z caller=http.go:261 level=warn traceID=6990622f6510bd17 msg="POST /push.v1.PusherService/Push (500) 2.893455ms Response: \"{\\\"code\\\":\\\"unknown\\\",\\\"message\\\":\\\"at least 1 live replicas required, could only find 0 - unhealthy instances: 172.24.0.2:4040\\\"}\" ws: false; Accept-Encoding: gzip; Connect-Protocol-Version: 1; Connect-Timeout-Ms: 9999; Content-Type: application/proto; User-Agent: GrafanaAgent/v0.41.1 (flow; linux; docker); X-Agent-Id: 58654f4e-73b6-4056-a9eb-c2da6d123399; "