search

jenningsloy318 / redfish_exporter

exporter to get metrics from redfish based hardware such as lenovo/dell/superc servers
Apache License 2.0
70 stars 62 forks source link

Error creating redfish client app=redfish_exporter #33

Closed tarekradi closed 2 years ago

tarekradi commented 2 years ago

``Hi. I am stuck.

Goal = Trying to use the redfish_exporter to load sensor data from BMC via redfish and track that sensor data in Prometheus.

Setup =

Problem = I keep seeing the following error error creating redfish client app=redfish_exporter error=Get "https://10.219.111.125/redfish/v1": proxyconnect tcp: tls: first record does not look like a TLS handshake target=10.219.111.125

Prometheus is running as a service. In /etc/prometheus/prometheus.yml i have the following section...

  - job_name: 'redfish-exporter'

    # metrics_path defaults to '/metrics'
    metrics_path: /redfish

    # scheme defaults to 'http'.

    static_configs:
    - targets:
       - 10.219.111.125
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9610  ### the address of the redfish-exporter address
      # (optional) when using group config add this to have group=my_group_name
      # target_label: __param_group
       # replacement: my_group_name

What am i doing wrong? Thank you for your help!

stmcginnis commented 2 years ago

What do you see if you point your browser at https://10.219.111.125/redfish/v1

tarekradi commented 2 years ago

From inside the host itself, if i point to https://10.219.111.125/redfish/v1 i get to connect "unable to connect". Same if i just say https://localhost/redfish/v1. Same if i point to that from a desktop that is able to reach http://10.219.111.125:9090 and see Prometheus.

Are IPs that i entered in my prometheus.yml and redfish_exporter config.yml correct?

For prometheus I set the "target" to .125, since that is the host OS in which the redfish_exporter is running, but for redfish_exporter I set the "host" to .126 since that is the BMC from which i need to pull the sensor data. Is that correct?

stmcginnis commented 2 years ago

Something is not configured right if the BMC address is .126. From that log, it is trying to access the redfish API (via the BMC) at the .125 address. Try setting "target" to .126.

stmcginnis commented 2 years ago

You should also be able to verify redfish is working by going to https://10.219.111.126/redfish/v1 from inside the host. That might be a good validation check just to make sure the service is operating correctly, though it does look like the main issue is just that it is going to the wrong IP when trying to access it.

tarekradi commented 2 years ago

thank you.

From inside the host, i am able to browse to 10.219.111.126/redfish/v1, but that should work regardless of prometheus and regardless of the redfish_exporter because i'm hitting the BMC directly that way. So it is expected to work and it is.

I updated the target in prometheus.yml to .126 and restarted the prometheus server. I also restarted redfish_exporter. I still see the following in the logs.

image

tarekradi commented 2 years ago

i just set no_proxy and ftp_proxy and socks_proxy. It seems to be working now. Can you please confirm?

When i browse to the redfish_exporter on the host and enter .126 as follows: image

I see the redfish_exporter logs give me the following... image

I don't see the TLS error. So maybe its working now? How can i check for sure?

tarekradi commented 2 years ago

What query do i type in Prometheus to see metrics of what's being scrapped? image

tarekradi commented 2 years ago

why should the "target" in prometheus.yml point to .126? That's the BMC address. Shouldn't it point to .125 (which is what i originally had), since .125 is where the redfish_exporter is running. The redfish_exporter would load telemetry from the BMC at .126.

I don't get how prometheus and redfish_exporter are expected to be connected after i applied your recommended change of setting the target to .126

stmcginnis commented 2 years ago

Sorry, you're probably right. Maybe someone else can jump in here with some suggestions. Just wanted to check since that error message seemed to indicate it was trying to connect to the host IP to access the redfish API rather than the BMC IP address.

tarekradi commented 2 years ago

Thanks. here is the current state i'm in...

The config.yml for redfish_exporter includes 10.219.111.126 (BMC) in the "hosts:" section. That's showing proper scraping as per screenshot i shared earlier

I updated prometheus.yml to point to .125 (host ip where redfish_exporter is running). Then restarted prometheus.

From the host's browser (i.e. on .125), i ran http://10.219.111.125/redfish?target=10.219.111.126 and it takes a while and loads. If i scroll to the bottom, i see "redfish_up 1". see below. So that looks like its working. image

However, prometheus is seeing 0 for redfish_up. What am i doing wrong here? Thanks image

jenningsloy318 commented 2 years ago

Hi,

I also felt strange, why promethus didn't call the real ip address of BMC, can you please post following things.

take this redfish config for example 

  - job_name: 'redfish-exporter'

    # metrics_path defaults to '/metrics'
    metrics_path: /redfish

    # scheme defaults to 'http'.

    static_configs:
    - targets:
       - 10.219.111.125
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9610  ### the address of the redfish-exporter address
      # (optional) when using group config add this to have group=my_group_name
      # target_label: __param_group
      - target_label: __address__
        replacement: localhost:9610  ### the address of the redfish-exporter address

as the comments pointed out, here is the ip address of redfish-export, hence the localhost should be replaced to the IP address of redfish-exporter

    static_configs:
    - targets:
       - 10.219.111.125

here is ip address of your server, which you are willing to monitor, so here you shouldn't add the ip address of redfish, but modified to bmc ip address

tarekradi commented 2 years ago

hi. Here are the 3 files in question. I have redacted some items and replaced with XXX Thank you for your help. 1) Redfish exporter config image

2) Prometheus config image

3) Logs Redfish_exporter log looks good.. image

Prometheus exporter log looks good image

Notice that i can run this command no problem from the browser http://10.219.111.125:9610/redfish?target=10.219.111.126&group= and i get redfish_up 1 image

Also, in prometheus, i see the following... image

However, redfish_up always shows a 0 as follows: image

As a reminder.. 10.219.111.125 is the OS, and 10.219.111.126 is the BMC... both are the same node.

jenningsloy318 commented 2 years ago

see my last comments to configure your prometheus， read the comments and the readme more carefully before you configure both prometheus and redfish.

here you misunderstand the addresses, please check prometheus config once more.

tarekradi commented 2 years ago

Regarding your last comment to configure prometheus, you said "here is ip address of your server, which you are willing to monitor, so here you shouldn't add the ip address of redfish, but modified to bmc ip address", but as I mentioned before, i tried both 125 and 126.

I went ahead again and changed this back to .126, restarted the prometheus service and restarted the redfish exporter just to be safe. I am still getting redfish_up = 0 in prometheus but I get 1 when i invoke the exporter through the web. What's worse is now the redfish_exporter is showing errors. See below.

image image image image

So far, i have provided you with all the details and i tried multiple settings, but the exporter and prometheus are not connecting together. Please advise.

jenningsloy318 commented 2 years ago

your logs and screenshot both show you still configure the monitored target to 10.219.111.125 which is the address of redfish, that is the problem, it should be 10.219.111.126

there must be wrong config at your promethus side, as I used this in my environment for years, no such issue

you can refer to my config here image and the content of tgroups/redfish-oob-targets.yml image

tarekradi commented 2 years ago

You said "your logs and screenshot both show you still configure the monitored target to 10.219.111.125 which is the address of redfish". 125 is not the address of redfish. 125 is the address of the unix box on which the redfish_exporter is running. 126 is the address of the BMC which is redfish-capable.

Earlier in this thread, i had the targets section point to 125. The redfish_exporter was not showing errors, but Prometheus was not able to see the exporter. Then you told me to change the targets section to point to 126, and i did. That's when the redfish_exporter started to show errors and prometheus still could not see the exporter.

Please tell me exactly how you recommend that configure this. Again, 125 is unix on which redfish_exporter is running and 126 is redfish-capable BMC.

Thanks.

jenningsloy318 commented 2 years ago

I said you should configure your BMC IP address in the stactic-configs section which is 126.

if your can post your screenshot regarding the promethus status-->targets page I have to say, there are a bunch of users used this tools, but none of them have such configuration problem, so firstly please read more carefully about the promethus documents

jenningsloy318 commented 2 years ago

I will close this issue, as it is really your configuration error and may not fully understand prometheus---> redfish-exporter-->monitor target model.

tarekradi commented 2 years ago

I resolved the issue. It was not my configuration. The Prometheus service was in a zombie state. I had to clean up the /var/lib/prometheus/lock file