Unable to connect grafana+prometeus

[99hops]

Can somebody please share how to connect the Grafana dashboard?

I've imported https://raw.githubusercontent.com/emeraldpay/dshackle/master/dashboard/dshackle.json and tried to add Prometeus data source to Grafana Enterprice with no luck.

monitoring:
  enabled: true
  jvm: false
  extended: false
  prometheus:
    enabled: true
    bind: 0.0.0.0
    port: 8000
    path: /status/prometheus

docker run -p 2449:2449 -p 8545:8545 -p 8000:8000 -v /etc/dshackle:/etc/dshackle emeraldpay/dshackle:0.12
...
MonitoringSetup | Run Prometheus metrics on 0.0.0.0:8000/status/prometheus

When I try to "save and test" (doing https://grafana-host/api/ds/query) I get different errors like so:

Error reading Prometheus: bad_response: readObjectStart: expect { or n, but found #, error found in #1 byte of ...|# HELP dsha|..., bigger context ...|# HELP dshackle_upstream_ws_fail_total Number of fa|...

Error reading Prometheus: bad_response: readObjectStart: expect { or n, but found #, error found in #1 byte of ...|# HELP dsha|..., bigger context ...|# HELP dshackle_request_grpc_request_total # TYPE|...

Still I can do curl http://127.0.0.1:8000/status/prometheus and get list of different metrics where I see lines starting with comments which to me seem related to the above response errors

dshackle_upstream_ws_conn_seconds_count{chain="ETH",upstream="node1",} 71.0
dshackle_upstream_ws_conn_seconds_sum{chain="ETH",upstream="node1",} 0.956933337
# HELP dshackle_upstream_ws_conn_seconds_max Request time through a WebSocket JSON RPC connection
# TYPE dshackle_upstream_ws_conn_seconds_max gauge
dshackle_upstream_ws_conn_seconds_max{chain="ETH",upstream="node2",} 0.375663177
dshackle_upstream_ws_conn_seconds_max{chain="ETH",upstream="node1",} 0.316129182
# HELP dshackle_upstreams_lag
# TYPE dshackle_upstreams_lag gauge
dshackle_upstreams_lag{chain="ETH",upstream="node2",} 1.0
dshackle_upstreams_lag{chain="ETH",upstream="node1",} 0.0
# HELP dshackle_upstreams_availability
# TYPE dshackle_upstreams_availability gauge

Am I doing something wrong here?

[splix]

Just to make sure you use it correctly. Do you have a Prometheus Server in your setup? From your message it does look like you try to connect Grafana to Dshackle directly. If that's what you do, it's not supposed to work that way. You need a separate instance of a Prometheus which is configured to gather metrics from Dshackle and connect Grafana to the Prometheus Server. Correct me if I misunderstood your report message.

[99hops]

My fault, there was no Prometheus instance indeed. I've added Prometheus and added scrape_config for Dshackle

  - job_name: dshackle
    static_configs:
      - targets: ['127.0.0.1:8000']

I am now able to connect Grafana to Prometheus and I can see Prometheus contains related data like dshackle_upstreams_availability (on the Prometheus web UI) but have no data on my "Dshackle Stats 2" dashboard.

What else needs to be added?

@splix Igor, please advice, sorry for asking but there is not much info on this topic.