grafana / pyroscope

Continuous Profiling Platform. Debug performance issues down to a single line of code
https://grafana.com/oss/pyroscope/
GNU Affero General Public License v3.0
10.13k stars 614 forks source link

magic ip: scheduler_processor and frontend_scheduler_worker #3617

Closed nexcode closed 1 month ago

nexcode commented 1 month ago

compose.yaml:

services:
  service:
    image: grafana/pyroscope:1.9.0
    command:
      - --config.file=/etc/pyroscope/pyroscope.yaml
    network_mode: host
    restart: on-failure
    volumes:
      - ./pyroscope.yaml:/etc/pyroscope/pyroscope.yaml:ro

pyroscope.yaml:

server:
  http_listen_address: 127.0.0.1
  http_listen_port: 3300
  grpc_listen_address: 127.0.0.1
  grpc_listen_port: 9395

frontend:
  address: 127.0.0.1

ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory

compactor:
  sharding_ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory

store_gateway:
  sharding_ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory

errors:

ts=2024-10-09T12:19:14.145052363Z caller=frontend_scheduler_worker.go:281 level=error component=frontend msg="error contacting scheduler" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 192.168.0.2:3300: connect: connection refused\"" addr=192.168.0.2:3300
ts=2024-10-09T12:19:14.927517886Z caller=scheduler_processor.go:125 level=error component=querier-worker msg="error processing requests from scheduler" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 192.168.0.2:3300: connect: connection refused\"" addr=192.168.0.2:3300

Question: Where is this address set (192.168.0.2)? I set all the fields I could find in the documentation to 127.0.0.1

kolesnikovae commented 1 month ago

I think a flag that overrides the default bind and advertised addresses could be really helpful here. For now, each component has to be configured explicitly:

-server.http-listen-address
-server.grpc-listen-address
-compactor.ring.instance-addr
-distributor.ring.instance-addr
-ingester.lifecycler.addr
-memberlist.advertise-addr
-memberlist.bind-addr
-overrides-exporter.ring.instance-addr
-query-frontend.instance-addr
-query-scheduler.ring.instance-addr
-store-gateway.sharding-ring.instance-addr

By default, the address is selected automatically based on the available network interfaces. This is because components are usually distributed and need to inform each other how to contact them

nexcode commented 1 month ago

This is not a declared option in the config doc (https://grafana.com/docs/pyroscope/latest/configure-server/reference-configuration-parameters/#query_scheduler):

query_scheduler:
  ring:
    instance_addr: 127.0.0.1

@kolesnikovae thanks for the tip!