Error reading Prometheus: An error occurred within the plugin

[Mr-Oca]

Is there an existing issue for this bug?

• [X] I have searched the existing issues before opening this bug report.

Current Behavior

I setup everything according to your tuto, but I try to us it on the grafana I host locally on docker. When I try to setup the data source, I add the ip "0.0.0.0:9100" (wich I can go to and read metrics) but when I try to save, I have thie error : "Error reading Prometheus: An error occurred within the plugin"

Expected Behavior

I should have a message saying "Data source is working".

Server type

Fabric

Server Version

1.19.3

Other Mods / Plugins in your server

Dynmap, fabric api, fabric carpet, fabric installer, fabric language kotlin, lithium, phosphore, sodium, unifiedmetrics

Error log

logger=data-proxy-log userId=1 orgId=1 uname=admin path=/api/datasources/proxy/2/api/v1/status/buildinfo remote_addr=192.168.1.10 referer=http://192.168.1.10:3000/datasources/edit/dNXlnQ04z/ t=2023-02-07T20:15:40.835278235Z level=error msg="Proxy request failed" err="dial tcp 0.0.0.0:9100: connect: connection refused"

logger=context userId=1 orgId=1 uname=admin t=2023-02-07T20:15:40.835341166Z level=error msg="Request Completed" method=GET path=/api/datasources/proxy/2/api/v1/status/buildinfo status=502 remote_addr=192.168.1.10 time_ms=1 duration=1.47335ms size=0 referer=http://192.168.1.10:3000/datasources/edit/dNXlnQ04z/ handler=/api/datasources/proxy/:id/*

logger=context userId=1 orgId=1 uname=admin t=2023-02-07T20:15:40.920678424Z level=error msg="Internal server error" error="[plugin.downstreamError] failed to query data: Post \"http://0.0.0.0:9100/api/v1/query\": dial tcp 0.0.0.0:9100: connect: connection refused" remote_addr=192.168.1.10 traceID=

logger=context userId=1 orgId=1 uname=admin t=2023-02-07T20:15:40.920728514Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=192.168.1.10 time_ms=1 duration=1.390618ms size=116 referer=http://192.168.1.10:3000/datasources/edit/dNXlnQ04z/ handler=/api/ds/query

More information

I don't know what to do to fix this. Can anyone help me ?

[Cubxity]

You cannot use 0.0.0.0 as the IP address of the scrape target. It is a special IP address used for listening only. Use 127.0.0.1 if you're hosting Prometheus on the same machine or whatever IP address your Minecraft server has.

[Mr-Oca]

Thank you for your answer. I changed the IP in Grafana and in the "agent.yml" file, so this is how I setup things :

• prometheus.yml

mode: "HTTP" # "HTTP" or "PUSHGATEWAY"

# Applies to "HTTP" mode.
http:
  host: "0.0.0.0"
  port: 9100 # The default in 0.4.x will be 9970.
  authentication:
    scheme: "NONE" # "NONE" or "BASIC"
    username: "username"
    password: "password"

# Applies to "PUSHGATEWAY" mode.
# Note that this is NOT compatible with remote_write.
pushGateway:
  job: "unifiedmetrics"
  url: "http://pushgateway:9091"
  authentication:
    scheme: "NONE" # "NONE" or "BASIC"
    username: "username"
    password: "password"
  interval: 10

• grafana docker-compose.yml

  
  version: "3"

services: agent: image: grafana/agent:latest restart: always container_name: agent security_opt:

• no-new-privileges:true volumes:
• ./agent.yml:/etc/agent-config/agent.yml entrypoint:
• /bin/agent
• -config.file=/etc/agent-config/agent.yml
• -prometheus.wal-directory=/tmp/agent/wal networks:
• prometheus

networks:

Change this to "pterodactyl_nw" and external to true IF you are using Pterodactyl

prometheus: external: true

- granafa agent

prometheus: global: scrape_interval: 5s configs:

• name: minecraft host_filter: false scrape_configs:
  • job_name: minecraft_scrape static_configs:

    Put your server's IP here

    - targets: [ "192.168.1.10:9100" ]
      # Feel free to add any labels here
      labels:
        node: "node-1"

    remote_write:

    Create an account at https://grafana.com/ and create an organization.

    Put Prometheus remote write credentials here. Make sure to update the push URL to match yours.

  • url: https://prometheus-us-central1.grafana.net/api/prom/push

    Create an API key with write scope on Grafana Cloud

    basic_auth: username: <instance id / remote_write username> password: <api key / remote_write password>

In grafana > data sources, I set "URL = http://192.168.1.10:9100/", but I don't change anything else.

Now I have the following as error :

logger=context userId=1 orgId=1 uname=admin t=2023-02-08T10:24:25.328919792Z level=error msg="Internal server error" error="[plugin.downstreamError] failed to query data: received empty response from prometheus" remote_addr=192.168.1.10 traceID=

logger=context userId=1 orgId=1 uname=admin t=2023-02-08T10:24:25.328995564Z level=error msg="Request Completed" method=POST path=/api/ds/query status=500 remote_addr=192.168.1.10 time_ms=3 duration=3.463041ms size=116 referer=http://192.168.1.10:3000/datasources/edit/dNXlnQ04z/ handler=/api/ds/query

So something changed, it seems that grafana CAN reach prometheus, but the response is empty.

[Cubxity]

You are using Grafana agent, which doesn't store metrics. It's used to push metrics to a remote Prometheus instance. In this case, you need to host a Prometheus server and point Grafana's datasource to it. You can't point Grafana directly to UnifiedMetrics.

[Mr-Oca]

Ah ok ! If someone is in the same trouble, I followed this video, just changed some stuff and voila ! Thanks for you answer !