Exporter scrape config error (or unclear docs?)

[keithwegner]

@derrick-dacosta thanks for offering to take a look.

I have a pretty simple Docker-Compose sandbox environment used to test various Prometheus exporters. While I have a number of exporters configured, I'm not quite sure what I am doing wrong with Fishymetrics' exporter. My hunch is that there's a lack of clarity in the README for the scrape_config, and/or that I've configured it incorrectly.

Here are the relevant files:

---

docker-compose.yaml

services:
  prometheus:
    image: prom/prometheus
    ports:
      - "9090:9090"
    ...

  redfish:
    image: dmtf/redfish-mockup-server:latest
    ports:
      - "8000:8000"

  fishymetrics_exporter:
    image: comcast/fishymetrics
    ports:
      - "9533:9533"
    environment:
      EXPORTER_PORT=9533
      BMC_USERNAME=admin
      BMC_PASSWORD=password
      BMC_TIMEOUT=15s
      BMC_SCHEME=http
      LOG_LEVEL=debug   

---

prometheus.yml

...
scrape_configs:
  - job_name: 'fishymetrics'
    static_configs:
      - targets: ['redfish:8000']
    metrics_path: /scrape

---

I can run the following example commands successfully:

• cURL the Redfish container's /redfish API from my development machine: curl localhost:8000/redfish/v1/Chassis
• cURL the Redfish container's /redfish API from a container on the same Docker network: docker run --rm --network sandbox_default curlimages/curl http://redfish:8000/redfish/v1/Chassis
• cURL the Fishymetrics container's /scrape API from a container on the same Docker network: docker run --rm --network sandbox_default curlmages/curl fishymetrics_exporter:9533/scrape?target=http://redfish:8000

However, the Fishymetrics container itself is not successfully scraping the Redfish Mockup Server container. Its logs show the following info and error messages (I'm not copy/pasting the but this should suffice):

"level":"info"..., "caller":"fishymetrics/main.go:134", "msg":"started scrape", "app":"fishymetrics", "host":"<fishymetrics_container_id>", "module":"", "module" : "", "model":"", "target":"http://redfish:8000/redfish/v1/Chassis", "credential_profile":"",...`
"level:"error",..., "caller":"exporter/exporter.go:180", "msg":"error when getting chassis url from ", "app":"fishymetrics", "host":"<fishymetrics_container_id>", "error":"HTTP status 404"...
"level:"error",..., "caller":"fishymetrics/main.go:170", "msg":"failed to create chassis exporter", "app":"fishymetrics", "host":"<fishymetrics_container_id>", "error":"HTTP status 404"...
"level:"info",...,  "caller":"fishymetrics/main.go:80",  "msg":"finished handling", "app":"fishymetrics", "host":"<fishymetrics_container_id>", "module":"", "target":"http://redfish:8000/redfish/v1/Chassis", "sourceAddr":"172.x.x.5:12345", "method":"GET", "url":"/scrape?target=http://redfish:8000/redfish/v1/Chassis", "proto":"HTTP/1.1", "status":500"...
"level:"error",..., "caller":"fishymetrics/main.go:134", "msg":"started scrape", "app":"fishymetrics", "host":"<fishymetrics_container_id>", "module":"", "model":"", "target":"http://redfish:8000/redfish/v1/Chassis/1U", "credential_profile":"",...
"level:"error",..., "caller":"fishymetrics/main.go:134", "msg":"started scrape","app":"fishymetrics", "host":"<fishymetrics_container_id>", "module":"", "model":"", "target":"http://redfish:8000/redfish/v1/Chassis/1U", "credential_profile":"",...

I noticed a whitespace character in the first error message: "error when getting chassis url from ", implying no URL is set? Hopefully that is a helpful clue.

[keithwegner]

OK, I discovered the issue.

It was due to the fact that the scrape_config's target did not begin with http://. Having a target of redfish:8000 is not sufficient; without explicitly adding http://, net/http's Transport.rountTrip method assumes the protocol is "redfish", which is invalid.

Similar issue / findings here:

• https://github.com/golang/go/issues/56027
• https://nanxiao.me/en/fix-unsupported-protocol-scheme-issue-in-golang/

I think the README's Prometheus Configuration section could use that specificity, as its example does not indicate this.

[derrick-dacosta]

Good find! I do agree that I need to improve the documentation by properly covering multiple edge cases.

Since you brought this to my attention I do want to make sure that a user can effectively set the BMC_SCHEME=https and then override it for individual hosts, via the prometheus config:

static_configs:
  - targets:
    - http://host1.example.com:8000
    - http://host2.example.com:8000
    - host3.example.com

In this case the 3rd host will be scraped as https://host3.example.com:443. As opposed to the first two which will be scraped as http://xxx:8000 etc. I can try to reproduce this locally and put in a fix if you like.