Unable to send metrics to Grafana Cloud

[dalareo]

Describe the bug Hi, we are trying to set up Prometheus metrics as explained in https://perfops.one/a-almost-free-observability-platform-part-2/ but we are not able to get metrics in Grafana Cloud.

To Reproduce Steps to reproduce the behavior:

1. Set Prometheus endpoint in Decalog Plugin. Wordpress is deployed inside Kubernetes cluster exposed by a service
2. Set Grafana Agent inside the cluster with the following config:

    metrics:
      wal_directory: /var/lib/agent/wal
      global:
        scrape_interval: 60s
        external_labels:
          cluster: 'clustername'
      configs:
      - name: integrations
        remote_write:
        - url: https://prometheus-prod-05-gb-south-0.grafana.net/api/prom/push
          basic_auth:
            username: 'username'
            password: 'password'
        scrape_configs:
        - job_name: decalog/scrapping
          static_configs:
            - targets: ['service-name']
              labels:
                process: 'agent'
          metrics_path: "/wp-json/decalog/v3/metrics?uuid=XXXXXX"

3. Go to grafana.com
4. Only scrape_duration_seconds metrics are shown with the selected labels

Expected behavior All metrics send by Decalog (those with wordpress_ prefix) are expected to be present. We are able to get Prometheus metrics in local Grafana using Push Gateway, but with Grafana Cloud we cannot use it.

Environment (please complete the following information):

• PHP version:PHP 7.4
• WordPress 6.1

Additional context Add any other context about the problem here.

[Pierre-Lannoy]

Hello @dalareo ! In targets you must specify the full url (and not path or something else) and it's seems it's not the case here. In the mentioned article, you have a screenshot of a working configuration… Could you just try to stick to it first, and we will see if there's other errors or problems after that. Thanks.

[dalareo]

targets field does not accept path nor scheme in its value, only hostname is allowed. If we set the full URL there we get:

2023/03/20 14:32:26 error loading config file /etc/agent/agent.yaml: "http://domain.com/wp-json/decalog/v3/metrics?uuid=44109443-f727-4d93-928f-f3f50b022c74" is not a valid hostname

Full URL is showing the expected metrics collection using the full with curl from grafana-agent or accessing through web browser.

[Pierre-Lannoy]

OK. So obviously, there's something wrong/outdated/deprecated in my article 😨 What if you add 127.0.0.1:80 as target?

[Pierre-Lannoy]

or whatever is your local ip/port…

[dalareo]

If I add a hostname (IP of servicename:port) no error is thrown but no metrics are shown in Grafana Cloud, only scrape_duration_seconds metrics

[Pierre-Lannoy]

OK. Let me some minutes to reinstall an agent on a test machine…

In the meantime, just to be sure, can you confirm the url /wp-json/decalog/v3/metrics?uuid=441… in your browser gives you metrics?

[dalareo]

Yes, it works like a charm in my browser. We can help with Php code also if you need to make some changes in the plugin!

[Pierre-Lannoy]

Thanks for the confirmation.

Thank you for your offer of help <3 Nevertheless, it is (at 99,9% chance) not a plugin issue but an agent config issue. That's why I will try to found what the hell has changed in the config…

[dalareo]

I've found the point:

- job_name: decalog/scrapping
          static_configs:
            - targets: ['service-name']
              labels:
                process: 'agent'
          metrics_path: "/wp-json/decalog/v3/metrics"
          params:
            uuid: ['uid-code]

[Pierre-Lannoy]

OMG! What you're saying is you can't have the query params in path, but you must specify it separately? And it is working now?

[dalareo]

Yes!! It works now. The parameter was allowed in the "metrics_path"

[Pierre-Lannoy]

OK. Cool. And good to know…