prometheus-community / ipmi_exporter

Remote IPMI exporter for Prometheus
MIT License
472 stars 133 forks source link

Error to collect dcmi #75

Open garciajrx opened 3 years ago

garciajrx commented 3 years ago

Using docker, version 1.4.0

Getting below when enable dcmi:

ime="2021-06-04T19:05:40Z" level=error msg="Failed to collect DCMI data from xx: error running ipmi-dcmi: exit status 1: ipmi_cmd_dcmi_get_power_reading: bad completion code\n" source="collector_dcmi.go:40" time="2021-06-04T19:05:47Z" level=error msg="Failed to collect DCMI data from xx: error running ipmi-dcmi: exit status 1: ipmi_cmd_dcmi_get_power_reading: bad completion code\n" source="collector_dcmi.go:40" time="2021-06-04T19:05:50Z" level=error msg="Failed to collect DCMI data from xx: error running ipmi-dcmi: exit status 1: ipmi_cmd_dcmi_get_power_reading: bad completion code\n" source="collector_dcmi.go:40" time="2021-06-04T19:06:50Z" level=error msg="Failed to collect DCMI data from xx: error running ipmi-dcmi: exit status 1: ipmi_cmd_dcmi_get_power_reading: bad completion code\n" source="collector_dcmi.go:40" time="2021-06-04T19:07:50Z" level=error msg="Failed to collect DCMI data from xx: error running ipmi-dcmi: exit status 1: ipmi_cmd_dcmi_get_power_reading: bad completion code\n" source="collector_dcmi.go:40" time="2021-06-04T19:08:50Z" level=error msg="Failed to collect DCMI data from xx: error running ipmi-dcmi: exit status 1: ipmi_cmd_dcmi_get_power_reading: bad completion code\n" source="collector_dcmi.go:40"

bitfehler commented 3 years ago

Thanks for reporting. Can you disclose what hardware you are running this against (i.e. the hardware you are monitoring)? Also, have you successfully extracted DCMI information from those machines with either ipmi-dcmi or other IPMI tools?

k0ste commented 2 years ago

This is caused by ipmi-dcmi exit code, not related to exporter actually I have a host with this problem

[root@mon]# ipmi-dcmi --config-file /tmp/test_data -h 172.16.18.26 --get-dcmi-capability-info
DCMI Specification Conformance                     : 1.5

Power Management / Monitoring Support              : Available
In-band System Interface Channel                   : Available
Serial TMODE                                       : Available
Out-Of-Band Secondary LAN Channel                  : Not present

Number of SEL entries                              : 1024
Record Level SEL Flush upon Rollover               : Not present
Entire SEL Flush upon Rollover                     : Available
SEL automatic rollover                             : Available
DHCP Host Name                                     : Not present
Asset Tag                                          : Not present
Sampling frequency for Temperature Monitoring      : Every 1 Second(s)

Power Management Device Slave Address              : 10h
Power Management Controller Device Revision        : 15
Power Management Controller Channel Number         : 15

Primary LAN Out-of-band Channel Number             : 1
Secondary LAN Out-of-band Channel Number           : Not supported
Serial Out-of-band TMODE Capability Channel Number : 2

Available Rolling Average Time Period              : 0 Seconds
Available Rolling Average Time Period              : 1 Hours
Available Rolling Average Time Period              : 1 Days
Available Rolling Average Time Period              : 7 Days

System power stat with debug:

172.16.18.26: =====================================================
172.16.18.26: IPMI 1.5 Get Channel Authentication Capabilities Request
172.16.18.26: =====================================================
172.16.18.26: RMCP Header:
172.16.18.26: ------------
172.16.18.26: [               6h] = version[ 8b]
172.16.18.26: [               0h] = reserved[ 8b]
172.16.18.26: [              FFh] = sequence_number[ 8b]
172.16.18.26: [               7h] = message_class.class[ 5b]
172.16.18.26: [               0h] = message_class.reserved[ 2b]
172.16.18.26: [               0h] = message_class.ack[ 1b]
172.16.18.26: IPMI Session Header:
172.16.18.26: --------------------
172.16.18.26: [               0h] = authentication_type[ 8b]
172.16.18.26: [               0h] = session_sequence_number[32b]
172.16.18.26: [               0h] = session_id[32b]
172.16.18.26: [               9h] = ipmi_msg_len[ 8b]
172.16.18.26: IPMI Message Header:
172.16.18.26: --------------------
172.16.18.26: [              20h] = rs_addr[ 8b]
172.16.18.26: [               0h] = rs_lun[ 2b]
172.16.18.26: [               6h] = net_fn[ 6b]
172.16.18.26: [              C8h] = checksum1[ 8b]
172.16.18.26: [              81h] = rq_addr[ 8b]
172.16.18.26: [               0h] = rq_lun[ 2b]
172.16.18.26: [              16h] = rq_seq[ 6b]
172.16.18.26: IPMI Command Data:
172.16.18.26: ------------------
172.16.18.26: [              38h] = cmd[ 8b]
172.16.18.26: [               Eh] = channel_number[ 4b]
172.16.18.26: [               0h] = reserved1[ 3b]
172.16.18.26: [               1h] = get_ipmi_v2.0_extended_data[ 1b]
172.16.18.26: [               2h] = maximum_privilege_level[ 4b]
172.16.18.26: [               0h] = reserved2[ 4b]
172.16.18.26: IPMI Trailer:
172.16.18.26: --------------
172.16.18.26: [              5Fh] = checksum2[ 8b]
172.16.18.26: =====================================================
172.16.18.26: IPMI 1.5 Get Channel Authentication Capabilities Response
172.16.18.26: =====================================================
172.16.18.26: RMCP Header:
172.16.18.26: ------------
172.16.18.26: [               6h] = version[ 8b]
172.16.18.26: [               0h] = reserved[ 8b]
172.16.18.26: [              FFh] = sequence_number[ 8b]
172.16.18.26: [               7h] = message_class.class[ 5b]
172.16.18.26: [               0h] = message_class.reserved[ 2b]
172.16.18.26: [               0h] = message_class.ack[ 1b]
172.16.18.26: IPMI Session Header:
172.16.18.26: --------------------
172.16.18.26: [               0h] = authentication_type[ 8b]
172.16.18.26: [               0h] = session_sequence_number[32b]
172.16.18.26: [               0h] = session_id[32b]
172.16.18.26: [              10h] = ipmi_msg_len[ 8b]
172.16.18.26: IPMI Message Header:
172.16.18.26: --------------------
172.16.18.26: [              81h] = rq_addr[ 8b]
172.16.18.26: [               0h] = rq_lun[ 2b]
172.16.18.26: [               7h] = net_fn[ 6b]
172.16.18.26: [              63h] = checksum1[ 8b]
172.16.18.26: [              20h] = rs_addr[ 8b]
172.16.18.26: [               0h] = rs_lun[ 2b]
172.16.18.26: [              16h] = rq_seq[ 6b]
172.16.18.26: IPMI Command Data:
172.16.18.26: ------------------
172.16.18.26: [              38h] = cmd[ 8b]
172.16.18.26: [               0h] = comp_code[ 8b]
172.16.18.26: [               1h] = channel_number[ 8b]
172.16.18.26: [               0h] = authentication_type.none[ 1b]
172.16.18.26: [               0h] = authentication_type.md2[ 1b]
172.16.18.26: [               1h] = authentication_type.md5[ 1b]
172.16.18.26: [               0h] = authentication_type.reserved1[ 1b]
172.16.18.26: [               0h] = authentication_type.straight_password_key[ 1b]
172.16.18.26: [               0h] = authentication_type.oem_prop[ 1b]
172.16.18.26: [               0h] = authentication_type.reserved2[ 1b]
172.16.18.26: [               1h] = authentication_type.ipmi_v2.0_extended_capabilities_available[ 1b]
172.16.18.26: [               0h] = authentication_status.anonymous_login[ 1b]
172.16.18.26: [               0h] = authentication_status.null_username[ 1b]
172.16.18.26: [               1h] = authentication_status.non_null_username[ 1b]
172.16.18.26: [               1h] = authentication_status.user_level_authentication[ 1b]
172.16.18.26: [               1h] = authentication_status.per_message_authentication[ 1b]
172.16.18.26: [               0h] = authentication_status.k_g[ 1b]
172.16.18.26: [               0h] = authentication_status.reserved[ 2b]
172.16.18.26: [               0h] = channel_supports_ipmi_v1.5_connections[ 1b]
172.16.18.26: [               1h] = channel_supports_ipmi_v2.0_connections[ 1b]
172.16.18.26: [               0h] = reserved[ 6b]
172.16.18.26: [               0h] = oem_id[24b]
172.16.18.26: [               0h] = oem_auxiliary_data[ 8b]
172.16.18.26: IPMI Trailer:
172.16.18.26: --------------
172.16.18.26: [              ADh] = checksum2[ 8b]
172.16.18.26: =====================================================
172.16.18.26: IPMI 2.0 Open Session Request
172.16.18.26: =====================================================
172.16.18.26: RMCP Header:
172.16.18.26: ------------
172.16.18.26: [               6h] = version[ 8b]
172.16.18.26: [               0h] = reserved[ 8b]
172.16.18.26: [              FFh] = sequence_number[ 8b]
172.16.18.26: [               7h] = message_class.class[ 5b]
172.16.18.26: [               0h] = message_class.reserved[ 2b]
172.16.18.26: [               0h] = message_class.ack[ 1b]
172.16.18.26: IPMI RMCPPLUS Session Header:
172.16.18.26: -----------------------------
172.16.18.26: [               6h] = authentication_type[ 4b]
172.16.18.26: [               0h] = reserved1[ 4b]
172.16.18.26: [              10h] = payload_type[ 6b]
172.16.18.26: [               0h] = payload_type.authenticated[ 1b]
172.16.18.26: [               0h] = payload_type.encrypted[ 1b]
172.16.18.26: [               0h] = session_id[32b]
172.16.18.26: [               0h] = session_sequence_number[32b]
172.16.18.26: [              20h] = ipmi_payload_len[16b]
172.16.18.26: IPMI RMCPPLUS Payload:
172.16.18.26: ----------------------
172.16.18.26: [  BYTE ARRAY ... ] = payload_data[32B]
172.16.18.26: [ 13h 00h 00h 00h 8Ah 01h 0Fh F9h ]
172.16.18.26: [ 00h 00h 00h 08h 01h 00h 00h 00h ]
172.16.18.26: [ 01h 00h 00h 08h 01h 00h 00h 00h ]
172.16.18.26: [ 02h 00h 00h 08h 01h 00h 00h 00h ]
172.16.18.26: IPMI Command Data:
172.16.18.26: ------------------
172.16.18.26: [              13h] = message_tag[ 8b]
172.16.18.26: [               0h] = requested_maximum_privilege_level[ 4b]
172.16.18.26: [               0h] = reserved1[ 4b]
172.16.18.26: [               0h] = reserved2[16b]
172.16.18.26: [        F90F018Ah] = remote_console_session_id[32b]
172.16.18.26: [               0h] = authentication_payload.payload_type[ 8b]
172.16.18.26: [               0h] = reserved3[16b]
172.16.18.26: [               8h] = authentication_payload.payload_length[ 8b]
172.16.18.26: [               1h] = authentication_payload.authentication_algorithm[ 6b]
172.16.18.26: [               0h] = reserved4[ 2b]
172.16.18.26: [               0h] = reserved5[24b]
172.16.18.26: [               1h] = integrity_payload.payload_type[ 8b]
172.16.18.26: [               0h] = reserved6[16b]
172.16.18.26: [               8h] = integrity_payload.payload_length[ 8b]
172.16.18.26: [               1h] = integrity_payload.integrity_algorithm[ 6b]
172.16.18.26: [               0h] = reserved7[ 2b]
172.16.18.26: [               0h] = reserved8[24b]
172.16.18.26: [               2h] = confidentiality_payload.payload_type[ 8b]
172.16.18.26: [               0h] = reserved9[16b]
172.16.18.26: [               8h] = confidentiality_payload.payload_length[ 8b]
172.16.18.26: [               1h] = confidentiality_payload.confidentiality_algorithm[ 6b]
172.16.18.26: [               0h] = reserved10[ 2b]
172.16.18.26: [               0h] = reserved11[24b]
172.16.18.26: =====================================================
172.16.18.26: IPMI 2.0 Open Session Response
172.16.18.26: =====================================================
172.16.18.26: RMCP Header:
172.16.18.26: ------------
172.16.18.26: [               6h] = version[ 8b]
172.16.18.26: [               0h] = reserved[ 8b]
172.16.18.26: [              FFh] = sequence_number[ 8b]
172.16.18.26: [               7h] = message_class.class[ 5b]
172.16.18.26: [               0h] = message_class.reserved[ 2b]
172.16.18.26: [               0h] = message_class.ack[ 1b]
172.16.18.26: IPMI RMCPPLUS Session Header:
172.16.18.26: -----------------------------
172.16.18.26: [               6h] = authentication_type[ 4b]
172.16.18.26: [               0h] = reserved1[ 4b]
172.16.18.26: [              11h] = payload_type[ 6b]
172.16.18.26: [               0h] = payload_type.authenticated[ 1b]
172.16.18.26: [               0h] = payload_type.encrypted[ 1b]
172.16.18.26: [               0h] = session_id[32b]
172.16.18.26: [               0h] = session_sequence_number[32b]
172.16.18.26: [              24h] = ipmi_payload_len[16b]
172.16.18.26: IPMI RMCPPLUS Payload:
172.16.18.26: ----------------------
172.16.18.26: [  BYTE ARRAY ... ] = payload_data[36B]
172.16.18.26: [ 13h 00h 04h 00h 8Ah 01h 0Fh F9h ]
172.16.18.26: [ 00h A6h 00h 02h 00h 00h 00h 08h ]
172.16.18.26: [ 01h 00h 00h 00h 01h 00h 00h 08h ]
172.16.18.26: [ 01h 00h 00h 00h 02h 00h 00h 08h ]
172.16.18.26: [ 01h 00h 00h 00h ]
172.16.18.26: IPMI Command Data:
172.16.18.26: ------------------
172.16.18.26: [              13h] = message_tag[ 8b]
172.16.18.26: [               0h] = rmcpplus_status_code[ 8b]
172.16.18.26: [               4h] = maximum_privilege_level[ 4b]
172.16.18.26: [               0h] = reserved1[ 4b]
172.16.18.26: [               0h] = reserved2[ 8b]
172.16.18.26: [        F90F018Ah] = remote_console_session_id[32b]
172.16.18.26: [         200A600h] = managed_system_session_id[32b]
172.16.18.26: [               0h] = authentication_payload.payload_type[ 8b]
172.16.18.26: [               0h] = reserved3[16b]
172.16.18.26: [               8h] = authentication_payload.payload_length[ 8b]
172.16.18.26: [               1h] = authentication_payload.authentication_algorithm[ 6b]
172.16.18.26: [               0h] = reserved4[ 2b]
172.16.18.26: [               0h] = reserved5[24b]
172.16.18.26: [               1h] = integrity_payload.payload_type[ 8b]
172.16.18.26: [               0h] = reserved6[16b]
172.16.18.26: [               8h] = integrity_payload.payload_length[ 8b]
172.16.18.26: [               1h] = integrity_payload.integrity_algorithm[ 6b]
172.16.18.26: [               0h] = reserved7[ 2b]
172.16.18.26: [               0h] = reserved8[24b]
172.16.18.26: [               2h] = confidentiality_payload.payload_type[ 8b]
172.16.18.26: [               0h] = reserved9[16b]
172.16.18.26: [               8h] = confidentiality_payload.payload_length[ 8b]
172.16.18.26: [               1h] = confidentiality_payload.confidentiality_algorithm[ 6b]
172.16.18.26: [               0h] = reserved10[ 2b]
172.16.18.26: [               0h] = reserved11[24b]
172.16.18.26: =====================================================
172.16.18.26: IPMI 2.0 RAKP Message 1 Request
172.16.18.26: =====================================================
172.16.18.26: RMCP Header:
172.16.18.26: ------------
172.16.18.26: [               6h] = version[ 8b]
172.16.18.26: [               0h] = reserved[ 8b]
172.16.18.26: [              FFh] = sequence_number[ 8b]
172.16.18.26: [               7h] = message_class.class[ 5b]
172.16.18.26: [               0h] = message_class.reserved[ 2b]
172.16.18.26: [               0h] = message_class.ack[ 1b]
172.16.18.26: IPMI RMCPPLUS Session Header:
172.16.18.26: -----------------------------
172.16.18.26: [               6h] = authentication_type[ 4b]
172.16.18.26: [               0h] = reserved1[ 4b]
172.16.18.26: [              12h] = payload_type[ 6b]
172.16.18.26: [               0h] = payload_type.authenticated[ 1b]
172.16.18.26: [               0h] = payload_type.encrypted[ 1b]
172.16.18.26: [               0h] = session_id[32b]
172.16.18.26: [               0h] = session_sequence_number[32b]
172.16.18.26: [              23h] = ipmi_payload_len[16b]
172.16.18.26: IPMI RMCPPLUS Payload:
172.16.18.26: ----------------------
172.16.18.26: [  BYTE ARRAY ... ] = payload_data[35B]
172.16.18.26: [ 14h 00h 00h 00h 00h A6h 00h 02h ]
172.16.18.26: [ BDh 8Eh EAh F7h 03h BAh 47h 4Ch ]
172.16.18.26: [ F7h 60h 19h B7h 3Bh 01h E6h 5Eh ]
172.16.18.26: [ 12h 00h 00h 07h 73h 68h 69h 6Eh ]
172.16.18.26: [ 6Bh 65h 6Eh ]
172.16.18.26: IPMI Command Data:
172.16.18.26: ------------------
172.16.18.26: [              14h] = message_tag[ 8b]
172.16.18.26: [               0h] = reserved1[24b]
172.16.18.26: [         200A600h] = managed_system_session_id[32b]
172.16.18.26: [  BYTE ARRAY ... ] = remote_console_random_number[16B]
172.16.18.26: [ BDh 8Eh EAh F7h 03h BAh 47h 4Ch ]
172.16.18.26: [ F7h 60h 19h B7h 3Bh 01h E6h 5Eh ]
172.16.18.26: [               2h] = requested_maximum_privilege_level[ 4b]
172.16.18.26: [               1h] = name_only_lookup[ 1b]
172.16.18.26: [               0h] = reserved2[ 3b]
172.16.18.26: [               0h] = reserved3[16b]
172.16.18.26: [               7h] = user_name_length[ 8b]
172.16.18.26: [  BYTE ARRAY ... ] = user_name[ 7B]
172.16.18.26: [ 73h 68h 69h 6Eh 6Bh 65h 6Eh ]
172.16.18.26: =====================================================
172.16.18.26: IPMI 2.0 RAKP Message 2 Response
172.16.18.26: =====================================================
172.16.18.26: RMCP Header:
172.16.18.26: ------------
172.16.18.26: [               6h] = version[ 8b]
172.16.18.26: [               0h] = reserved[ 8b]
172.16.18.26: [              FFh] = sequence_number[ 8b]
172.16.18.26: [               7h] = message_class.class[ 5b]
172.16.18.26: [               0h] = message_class.reserved[ 2b]
172.16.18.26: [               0h] = message_class.ack[ 1b]
172.16.18.26: IPMI RMCPPLUS Session Header:
172.16.18.26: -----------------------------
172.16.18.26: [               6h] = authentication_type[ 4b]
172.16.18.26: [               0h] = reserved1[ 4b]
172.16.18.26: [              13h] = payload_type[ 6b]
172.16.18.26: [               0h] = payload_type.authenticated[ 1b]
172.16.18.26: [               0h] = payload_type.encrypted[ 1b]
172.16.18.26: [               0h] = session_id[32b]
172.16.18.26: [               0h] = session_sequence_number[32b]
172.16.18.26: [              3Ch] = ipmi_payload_len[16b]
172.16.18.26: IPMI RMCPPLUS Payload:
172.16.18.26: ----------------------
172.16.18.26: [  BYTE ARRAY ... ] = payload_data[60B]
172.16.18.26: [ 14h 00h 00h 00h 8Ah 01h 0Fh F9h ]
172.16.18.26: [ 22h 0Fh B1h 99h F9h 05h 10h 1Eh ]
172.16.18.26: [ 17h 2Dh 7Ch 25h 20h 26h 60h 91h ]
172.16.18.26: [ 08h 00h 00h 00h 00h 00h 00h 00h ]
172.16.18.26: [ 00h 00h 00h 00h 00h 00h 00h 00h ]
172.16.18.26: [ 2Fh D6h F5h FAh C5h 99h 5Fh 21h ]
172.16.18.26: [ 74h D4h C5h 02h 41h 4Fh 44h 3Ah ]
172.16.18.26: [ 49h 3Ch D1h 88h ]
172.16.18.26: IPMI Command Data:
172.16.18.26: ------------------
172.16.18.26: [              14h] = message_tag[ 8b]
172.16.18.26: [               0h] = rmcpplus_status_code[ 8b]
172.16.18.26: [               0h] = reserved1[16b]
172.16.18.26: [        F90F018Ah] = remote_console_session_id[32b]
172.16.18.26: [  BYTE ARRAY ... ] = managed_system_random_number[16B]
172.16.18.26: [ 22h 0Fh B1h 99h F9h 05h 10h 1Eh ]
172.16.18.26: [ 17h 2Dh 7Ch 25h 20h 26h 60h 91h ]
172.16.18.26: [  BYTE ARRAY ... ] = managed_system_guid[16B]
172.16.18.26: [ 08h 00h 00h 00h 00h 00h 00h 00h ]
172.16.18.26: [ 00h 00h 00h 00h 00h 00h 00h 00h ]
172.16.18.26: [  BYTE ARRAY ... ] = key_exchange_authentication_code[20B]
172.16.18.26: [ 2Fh D6h F5h FAh C5h 99h 5Fh 21h ]
172.16.18.26: [ 74h D4h C5h 02h 41h 4Fh 44h 3Ah ]
172.16.18.26: [ 49h 3Ch D1h 88h ]
172.16.18.26: =====================================================
172.16.18.26: IPMI 2.0 RAKP Message 3 Request
172.16.18.26: =====================================================
172.16.18.26: RMCP Header:
172.16.18.26: ------------
172.16.18.26: [               6h] = version[ 8b]
172.16.18.26: [               0h] = reserved[ 8b]
172.16.18.26: [              FFh] = sequence_number[ 8b]
172.16.18.26: [               7h] = message_class.class[ 5b]
172.16.18.26: [               0h] = message_class.reserved[ 2b]
172.16.18.26: [               0h] = message_class.ack[ 1b]
172.16.18.26: IPMI RMCPPLUS Session Header:
172.16.18.26: -----------------------------
172.16.18.26: [               6h] = authentication_type[ 4b]
172.16.18.26: [               0h] = reserved1[ 4b]
172.16.18.26: [              14h] = payload_type[ 6b]
172.16.18.26: [               0h] = payload_type.authenticated[ 1b]
172.16.18.26: [               0h] = payload_type.encrypted[ 1b]
172.16.18.26: [               0h] = session_id[32b]
172.16.18.26: [               0h] = session_sequence_number[32b]
172.16.18.26: [              1Ch] = ipmi_payload_len[16b]
172.16.18.26: IPMI RMCPPLUS Payload:
172.16.18.26: ----------------------
172.16.18.26: [  BYTE ARRAY ... ] = payload_data[28B]
172.16.18.26: [ 15h 00h 00h 00h 00h A6h 00h 02h ]
172.16.18.26: [ ACh 62h 33h C0h ADh D0h C0h 0Eh ]
172.16.18.26: [ 1Eh 07h 07h 24h D7h 36h F2h A4h ]
172.16.18.26: [ 7Ah 1Eh F7h 16h ]
172.16.18.26: IPMI Command Data:
172.16.18.26: ------------------
172.16.18.26: [              15h] = message_tag[ 8b]
172.16.18.26: [               0h] = rmcpplus_status_code[ 8b]
172.16.18.26: [               0h] = reserved1[16b]
172.16.18.26: [         200A600h] = managed_system_session_id[32b]
172.16.18.26: [  BYTE ARRAY ... ] = key_exchange_authentication_code[20B]
172.16.18.26: [ ACh 62h 33h C0h ADh D0h C0h 0Eh ]
172.16.18.26: [ 1Eh 07h 07h 24h D7h 36h F2h A4h ]
172.16.18.26: [ 7Ah 1Eh F7h 16h ]
172.16.18.26: =====================================================
172.16.18.26: IPMI 2.0 RAKP Message 4 Response
172.16.18.26: =====================================================
172.16.18.26: RMCP Header:
172.16.18.26: ------------
172.16.18.26: [               6h] = version[ 8b]
172.16.18.26: [               0h] = reserved[ 8b]
172.16.18.26: [              FFh] = sequence_number[ 8b]
172.16.18.26: [               7h] = message_class.class[ 5b]
172.16.18.26: [               0h] = message_class.reserved[ 2b]
172.16.18.26: [               0h] = message_class.ack[ 1b]
172.16.18.26: IPMI RMCPPLUS Session Header:
172.16.18.26: -----------------------------
172.16.18.26: [               6h] = authentication_type[ 4b]
172.16.18.26: [               0h] = reserved1[ 4b]
172.16.18.26: [              15h] = payload_type[ 6b]
172.16.18.26: [               0h] = payload_type.authenticated[ 1b]
172.16.18.26: [               0h] = payload_type.encrypted[ 1b]
172.16.18.26: [               0h] = session_id[32b]
172.16.18.26: [               0h] = session_sequence_number[32b]
172.16.18.26: [              14h] = ipmi_payload_len[16b]
172.16.18.26: IPMI RMCPPLUS Payload:
172.16.18.26: ----------------------
172.16.18.26: [  BYTE ARRAY ... ] = payload_data[20B]
172.16.18.26: [ 15h 00h 00h 00h 8Ah 01h 0Fh F9h ]
172.16.18.26: [ 82h 28h 94h CCh E9h 01h FAh 63h ]
172.16.18.26: [ 88h 9Bh B1h BCh ]
172.16.18.26: IPMI Command Data:
172.16.18.26: ------------------
172.16.18.26: [              15h] = message_tag[ 8b]
172.16.18.26: [               0h] = rmcpplus_status_code[ 8b]
172.16.18.26: [               0h] = reserved1[16b]
172.16.18.26: [        F90F018Ah] = remote_console_session_id[32b]
172.16.18.26: [  BYTE ARRAY ... ] = integrity_check_value[12B]
172.16.18.26: [ 82h 28h 94h CCh E9h 01h FAh 63h ]
172.16.18.26: [ 88h 9Bh B1h BCh ]
172.16.18.26: =====================================================
172.16.18.26: IPMI 2.0 Set Session Privilege Level Request
172.16.18.26: =====================================================
172.16.18.26: RMCP Header:
172.16.18.26: ------------
172.16.18.26: [               6h] = version[ 8b]
172.16.18.26: [               0h] = reserved[ 8b]
172.16.18.26: [              FFh] = sequence_number[ 8b]
172.16.18.26: [               7h] = message_class.class[ 5b]
172.16.18.26: [               0h] = message_class.reserved[ 2b]
172.16.18.26: [               0h] = message_class.ack[ 1b]
172.16.18.26: IPMI RMCPPLUS Session Header:
172.16.18.26: -----------------------------
172.16.18.26: [               6h] = authentication_type[ 4b]
172.16.18.26: [               0h] = reserved1[ 4b]
172.16.18.26: [               0h] = payload_type[ 6b]
172.16.18.26: [               1h] = payload_type.authenticated[ 1b]
172.16.18.26: [               1h] = payload_type.encrypted[ 1b]
172.16.18.26: [         200A600h] = session_id[32b]
172.16.18.26: [               1h] = session_sequence_number[32b]
172.16.18.26: [              20h] = ipmi_payload_len[16b]
172.16.18.26: IPMI RMCPPLUS Payload:
172.16.18.26: ----------------------
172.16.18.26: [  BYTE ARRAY ... ] = confidentiality_header[16B]
172.16.18.26: [ 1Ah 91h 7Bh 36h 84h B1h 19h 8Ch ]
172.16.18.26: [ BFh 86h 02h F0h 0Bh FEh 1Ah 6Fh ]
172.16.18.26: [  BYTE ARRAY ... ] = payload_data[ 8B]
172.16.18.26: [ 20h 18h C8h 81h 5Ch 3Bh 02h E6h ]
172.16.18.26: [ 707060504030201h] = confidentiality_trailer[64b]
172.16.18.26: IPMI Message Header:
172.16.18.26: --------------------
172.16.18.26: [              20h] = rs_addr[ 8b]
172.16.18.26: [               0h] = rs_lun[ 2b]
172.16.18.26: [               6h] = net_fn[ 6b]
172.16.18.26: [              C8h] = checksum1[ 8b]
172.16.18.26: [              81h] = rq_addr[ 8b]
172.16.18.26: [               0h] = rq_lun[ 2b]
172.16.18.26: [              17h] = rq_seq[ 6b]
172.16.18.26: IPMI Command Data:
172.16.18.26: ------------------
172.16.18.26: [              3Bh] = cmd[ 8b]
172.16.18.26: [               2h] = privilege_level[ 4b]
172.16.18.26: [               0h] = reserved1[ 4b]
172.16.18.26: IPMI Trailer:
172.16.18.26: -------------
172.16.18.26: [              E6h] = checksum2[ 8b]
172.16.18.26: IPMI RMCPPLUS Session Trailer:
172.16.18.26: ------------------------------
172.16.18.26: [            FFFFh] = integrity_pad[16b]
172.16.18.26: [               2h] = pad_length[ 8b]
172.16.18.26: [               7h] = next_header[ 8b]
172.16.18.26: [  BYTE ARRAY ... ] = authentication_code[12B]
172.16.18.26: [ C2h 9Eh BDh 5Ch 7Dh 81h D8h 7Bh ]
172.16.18.26: [ 2Fh 1Ch 0Ah 74h ]
172.16.18.26: =====================================================
172.16.18.26: IPMI 2.0 Set Session Privilege Level Response
172.16.18.26: =====================================================
172.16.18.26: RMCP Header:
172.16.18.26: ------------
172.16.18.26: [               6h] = version[ 8b]
172.16.18.26: [               0h] = reserved[ 8b]
172.16.18.26: [              FFh] = sequence_number[ 8b]
172.16.18.26: [               7h] = message_class.class[ 5b]
172.16.18.26: [               0h] = message_class.reserved[ 2b]
172.16.18.26: [               0h] = message_class.ack[ 1b]
172.16.18.26: IPMI RMCPPLUS Session Header:
172.16.18.26: -----------------------------
172.16.18.26: [               6h] = authentication_type[ 4b]
172.16.18.26: [               0h] = reserved1[ 4b]
172.16.18.26: [               0h] = payload_type[ 6b]
172.16.18.26: [               1h] = payload_type.authenticated[ 1b]
172.16.18.26: [               1h] = payload_type.encrypted[ 1b]
172.16.18.26: [        F90F018Ah] = session_id[32b]
172.16.18.26: [               1h] = session_sequence_number[32b]
172.16.18.26: [              20h] = ipmi_payload_len[16b]
172.16.18.26: IPMI RMCPPLUS Payload:
172.16.18.26: ----------------------
172.16.18.26: [  BYTE ARRAY ... ] = confidentiality_header[16B]
172.16.18.26: [ 22h 0Fh B1h 99h F9h 05h 10h 1Eh ]
172.16.18.26: [ 17h 2Dh 7Ch 25h 20h 26h 60h 91h ]
172.16.18.26: [  BYTE ARRAY ... ] = payload_data[ 9B]
172.16.18.26: [ 81h 1Ch 63h 20h 5Ch 3Bh 00h 02h ]
172.16.18.26: [ 47h ]
172.16.18.26: [   6060504030201h] = confidentiality_trailer[56b]
172.16.18.26: IPMI Message Header:
172.16.18.26: --------------------
172.16.18.26: [              81h] = rq_addr[ 8b]
172.16.18.26: [               0h] = rq_lun[ 2b]
172.16.18.26: [               7h] = net_fn[ 6b]
172.16.18.26: [              63h] = checksum1[ 8b]
172.16.18.26: [              20h] = rs_addr[ 8b]
172.16.18.26: [               0h] = rs_lun[ 2b]
172.16.18.26: [              17h] = rq_seq[ 6b]
172.16.18.26: IPMI Command Data:
172.16.18.26: ------------------
172.16.18.26: [              3Bh] = cmd[ 8b]
172.16.18.26: [               0h] = comp_code[ 8b]
172.16.18.26: [               2h] = privilege_level[ 4b]
172.16.18.26: [               0h] = reserved1[ 4b]
172.16.18.26: IPMI Trailer:
172.16.18.26: -------------
172.16.18.26: [              47h] = checksum2[ 8b]
172.16.18.26: IPMI RMCPPLUS Session Trailer:
172.16.18.26: ------------------------------
172.16.18.26: [            FFFFh] = integrity_pad[16b]
172.16.18.26: [               2h] = pad_length[ 8b]
172.16.18.26: [               7h] = next_header[ 8b]
172.16.18.26: [  BYTE ARRAY ... ] = authentication_code[12B]
172.16.18.26: [ 16h E9h BCh 0Bh 05h 53h 77h 78h ]
172.16.18.26: [ CCh 12h DCh 3Eh ]
172.16.18.26: =====================================================
172.16.18.26: IPMI 2.0 Group Extension - Get Power Reading Request
172.16.18.26: =====================================================
172.16.18.26: RMCP Header:
172.16.18.26: ------------
172.16.18.26: [               6h] = version[ 8b]
172.16.18.26: [               0h] = reserved[ 8b]
172.16.18.26: [              FFh] = sequence_number[ 8b]
172.16.18.26: [               7h] = message_class.class[ 5b]
172.16.18.26: [               0h] = message_class.reserved[ 2b]
172.16.18.26: [               0h] = message_class.ack[ 1b]
172.16.18.26: IPMI RMCPPLUS Session Header:
172.16.18.26: -----------------------------
172.16.18.26: [               6h] = authentication_type[ 4b]
172.16.18.26: [               0h] = reserved1[ 4b]
172.16.18.26: [               0h] = payload_type[ 6b]
172.16.18.26: [               1h] = payload_type.authenticated[ 1b]
172.16.18.26: [               1h] = payload_type.encrypted[ 1b]
172.16.18.26: [         200A600h] = session_id[32b]
172.16.18.26: [               2h] = session_sequence_number[32b]
172.16.18.26: [              20h] = ipmi_payload_len[16b]
172.16.18.26: IPMI RMCPPLUS Payload:
172.16.18.26: ----------------------
172.16.18.26: [  BYTE ARRAY ... ] = confidentiality_header[16B]
172.16.18.26: [ 3Fh E5h EFh 6Bh 9Bh 8Ah A1h 8Bh ]
172.16.18.26: [ 06h 3Eh 88h 3Ah 1Ah C7h A7h A8h ]
172.16.18.26: [  BYTE ARRAY ... ] = payload_data[11B]
172.16.18.26: [ 20h B0h 30h 81h 60h 02h DCh 01h ]
172.16.18.26: [ 00h 00h 40h ]
172.16.18.26: [       404030201h] = confidentiality_trailer[40b]
172.16.18.26: IPMI Message Header:
172.16.18.26: --------------------
172.16.18.26: [              20h] = rs_addr[ 8b]
172.16.18.26: [               0h] = rs_lun[ 2b]
172.16.18.26: [              2Ch] = net_fn[ 6b]
172.16.18.26: [              30h] = checksum1[ 8b]
172.16.18.26: [              81h] = rq_addr[ 8b]
172.16.18.26: [               0h] = rq_lun[ 2b]
172.16.18.26: [              18h] = rq_seq[ 6b]
172.16.18.26: IPMI Command Data:
172.16.18.26: ------------------
172.16.18.26: [               2h] = cmd[ 8b]
172.16.18.26: [              DCh] = group_extension_identification[ 8b]
172.16.18.26: [               1h] = mode[ 8b]
172.16.18.26: [               0h] = mode_attributes[ 8b]
172.16.18.26: [               0h] = reserved[ 8b]
172.16.18.26: IPMI Trailer:
172.16.18.26: -------------
172.16.18.26: [              40h] = checksum2[ 8b]
172.16.18.26: IPMI RMCPPLUS Session Trailer:
172.16.18.26: ------------------------------
172.16.18.26: [            FFFFh] = integrity_pad[16b]
172.16.18.26: [               2h] = pad_length[ 8b]
172.16.18.26: [               7h] = next_header[ 8b]
172.16.18.26: [  BYTE ARRAY ... ] = authentication_code[12B]
172.16.18.26: [ A1h 5Ah BEh 50h FCh 25h 88h 98h ]
172.16.18.26: [ 51h 16h D2h 7Ah ]
172.16.18.26: =====================================================
172.16.18.26: IPMI 2.0 Group Extension - Get Power Reading Response
172.16.18.26: =====================================================
172.16.18.26: RMCP Header:
172.16.18.26: ------------
172.16.18.26: [               6h] = version[ 8b]
172.16.18.26: [               0h] = reserved[ 8b]
172.16.18.26: [              FFh] = sequence_number[ 8b]
172.16.18.26: [               7h] = message_class.class[ 5b]
172.16.18.26: [               0h] = message_class.reserved[ 2b]
172.16.18.26: [               0h] = message_class.ack[ 1b]
172.16.18.26: IPMI RMCPPLUS Session Header:
172.16.18.26: -----------------------------
172.16.18.26: [               6h] = authentication_type[ 4b]
172.16.18.26: [               0h] = reserved1[ 4b]
172.16.18.26: [               0h] = payload_type[ 6b]
172.16.18.26: [               1h] = payload_type.authenticated[ 1b]
172.16.18.26: [               1h] = payload_type.encrypted[ 1b]
172.16.18.26: [        F90F018Ah] = session_id[32b]
172.16.18.26: [               2h] = session_sequence_number[32b]
172.16.18.26: [              20h] = ipmi_payload_len[16b]
172.16.18.26: IPMI RMCPPLUS Payload:
172.16.18.26: ----------------------
172.16.18.26: [  BYTE ARRAY ... ] = confidentiality_header[16B]
172.16.18.26: [ 22h 0Fh B1h 99h F9h 05h 10h 1Eh ]
172.16.18.26: [ 17h 2Dh 7Ch 25h 20h 26h 60h 91h ]
172.16.18.26: [  BYTE ARRAY ... ] = payload_data[ 8B]
172.16.18.26: [ 81h B4h CBh 20h 60h 02h D5h A9h ]
172.16.18.26: [ 707060504030201h] = confidentiality_trailer[64b]
172.16.18.26: IPMI Message Header:
172.16.18.26: --------------------
172.16.18.26: [              81h] = rq_addr[ 8b]
172.16.18.26: [               0h] = rq_lun[ 2b]
172.16.18.26: [              2Dh] = net_fn[ 6b]
172.16.18.26: [              CBh] = checksum1[ 8b]
172.16.18.26: [              20h] = rs_addr[ 8b]
172.16.18.26: [               0h] = rs_lun[ 2b]
172.16.18.26: [              18h] = rq_seq[ 6b]
172.16.18.26: IPMI Command Data:
172.16.18.26: ------------------
172.16.18.26: [               2h] = cmd[ 8b]
172.16.18.26: [              D5h] = comp_code[ 8b]
172.16.18.26: IPMI Trailer:
172.16.18.26: -------------
172.16.18.26: [              A9h] = checksum2[ 8b]
172.16.18.26: IPMI RMCPPLUS Session Trailer:
172.16.18.26: ------------------------------
172.16.18.26: [            FFFFh] = integrity_pad[16b]
172.16.18.26: [               2h] = pad_length[ 8b]
172.16.18.26: [               7h] = next_header[ 8b]
172.16.18.26: [  BYTE ARRAY ... ] = authentication_code[12B]
172.16.18.26: [ E0h BBh C1h D0h A9h E1h 97h 6Eh ]
172.16.18.26: [ A4h 6Fh 8Bh 81h ]
ipmi_cmd_dcmi_get_power_reading: bad completion code
172.16.18.26: =====================================================
172.16.18.26: IPMI 2.0 Close Session Request
172.16.18.26: =====================================================
172.16.18.26: RMCP Header:
172.16.18.26: ------------
172.16.18.26: [               6h] = version[ 8b]
172.16.18.26: [               0h] = reserved[ 8b]
172.16.18.26: [              FFh] = sequence_number[ 8b]
172.16.18.26: [               7h] = message_class.class[ 5b]
172.16.18.26: [               0h] = message_class.reserved[ 2b]
172.16.18.26: [               0h] = message_class.ack[ 1b]
172.16.18.26: IPMI RMCPPLUS Session Header:
172.16.18.26: -----------------------------
172.16.18.26: [               6h] = authentication_type[ 4b]
172.16.18.26: [               0h] = reserved1[ 4b]
172.16.18.26: [               0h] = payload_type[ 6b]
172.16.18.26: [               1h] = payload_type.authenticated[ 1b]
172.16.18.26: [               1h] = payload_type.encrypted[ 1b]
172.16.18.26: [         200A600h] = session_id[32b]
172.16.18.26: [               3h] = session_sequence_number[32b]
172.16.18.26: [              20h] = ipmi_payload_len[16b]
172.16.18.26: IPMI RMCPPLUS Payload:
172.16.18.26: ----------------------
172.16.18.26: [  BYTE ARRAY ... ] = confidentiality_header[16B]
172.16.18.26: [ 1Ah E2h FAh 3Dh EDh 07h A6h 5Fh ]
172.16.18.26: [ 79h 78h EDh A3h C8h 63h 0Ah 29h ]
172.16.18.26: [  BYTE ARRAY ... ] = payload_data[11B]
172.16.18.26: [ 20h 18h C8h 81h 64h 3Ch 00h A6h ]
172.16.18.26: [ 00h 02h 37h ]
172.16.18.26: [       404030201h] = confidentiality_trailer[40b]
172.16.18.26: IPMI Message Header:
172.16.18.26: --------------------
172.16.18.26: [              20h] = rs_addr[ 8b]
172.16.18.26: [               0h] = rs_lun[ 2b]
172.16.18.26: [               6h] = net_fn[ 6b]
172.16.18.26: [              C8h] = checksum1[ 8b]
172.16.18.26: [              81h] = rq_addr[ 8b]
172.16.18.26: [               0h] = rq_lun[ 2b]
172.16.18.26: [              19h] = rq_seq[ 6b]
172.16.18.26: IPMI Command Data:
172.16.18.26: ------------------
172.16.18.26: [              3Ch] = cmd[ 8b]
172.16.18.26: [         200A600h] = session_id[32b]
172.16.18.26: IPMI Trailer:
172.16.18.26: -------------
172.16.18.26: [              37h] = checksum2[ 8b]
172.16.18.26: IPMI RMCPPLUS Session Trailer:
172.16.18.26: ------------------------------
172.16.18.26: [            FFFFh] = integrity_pad[16b]
172.16.18.26: [               2h] = pad_length[ 8b]
172.16.18.26: [               7h] = next_header[ 8b]
172.16.18.26: [  BYTE ARRAY ... ] = authentication_code[12B]
172.16.18.26: [ 07h A0h 60h 7Dh C0h 02h 27h 1Bh ]
172.16.18.26: [ 25h F1h 00h 4Eh ]
172.16.18.26: =====================================================
172.16.18.26: IPMI 2.0 Close Session Response
172.16.18.26: =====================================================
172.16.18.26: RMCP Header:
172.16.18.26: ------------
172.16.18.26: [               6h] = version[ 8b]
172.16.18.26: [               0h] = reserved[ 8b]
172.16.18.26: [              FFh] = sequence_number[ 8b]
172.16.18.26: [               7h] = message_class.class[ 5b]
172.16.18.26: [               0h] = message_class.reserved[ 2b]
172.16.18.26: [               0h] = message_class.ack[ 1b]
172.16.18.26: IPMI RMCPPLUS Session Header:
172.16.18.26: -----------------------------
172.16.18.26: [               6h] = authentication_type[ 4b]
172.16.18.26: [               0h] = reserved1[ 4b]
172.16.18.26: [               0h] = payload_type[ 6b]
172.16.18.26: [               1h] = payload_type.authenticated[ 1b]
172.16.18.26: [               1h] = payload_type.encrypted[ 1b]
172.16.18.26: [        F90F018Ah] = session_id[32b]
172.16.18.26: [               3h] = session_sequence_number[32b]
172.16.18.26: [              20h] = ipmi_payload_len[16b]
172.16.18.26: IPMI RMCPPLUS Payload:
172.16.18.26: ----------------------
172.16.18.26: [  BYTE ARRAY ... ] = confidentiality_header[16B]
172.16.18.26: [ 22h 0Fh B1h 99h F9h 05h 10h 1Eh ]
172.16.18.26: [ 17h 2Dh 7Ch 25h 20h 26h 60h 91h ]
172.16.18.26: [  BYTE ARRAY ... ] = payload_data[ 8B]
172.16.18.26: [ 81h 1Ch 63h 20h 64h 3Ch 00h 40h ]
172.16.18.26: [ 707060504030201h] = confidentiality_trailer[64b]
172.16.18.26: IPMI Message Header:
172.16.18.26: --------------------
172.16.18.26: [              81h] = rq_addr[ 8b]
172.16.18.26: [               0h] = rq_lun[ 2b]
172.16.18.26: [               7h] = net_fn[ 6b]
172.16.18.26: [              63h] = checksum1[ 8b]
172.16.18.26: [              20h] = rs_addr[ 8b]
172.16.18.26: [               0h] = rs_lun[ 2b]
172.16.18.26: [              19h] = rq_seq[ 6b]
172.16.18.26: IPMI Command Data:
172.16.18.26: ------------------
172.16.18.26: [              3Ch] = cmd[ 8b]
172.16.18.26: [               0h] = comp_code[ 8b]
172.16.18.26: IPMI Trailer:
172.16.18.26: -------------
172.16.18.26: [              40h] = checksum2[ 8b]
172.16.18.26: IPMI RMCPPLUS Session Trailer:
172.16.18.26: ------------------------------
172.16.18.26: [            FFFFh] = integrity_pad[16b]
172.16.18.26: [               2h] = pad_length[ 8b]
172.16.18.26: [               7h] = next_header[ 8b]
172.16.18.26: [  BYTE ARRAY ... ] = authentication_code[12B]
172.16.18.26: [ 5Bh 31h 7Eh 93h 72h D3h DEh 2Eh ]
172.16.18.26: [ EDh 6Dh 54h D1h ]
mtds commented 1 year ago

@bitfehler I am not the OP but we are facing the same issue. As reported above, the problem is less with the exporter itself and mostly on the ipmi-dcmi side and its way of reporting the error. Biggest issue is that those error messages pile up pretty quickly and our logs becomes very 'noisy'.

Since you have asked the following questions, I can give the following information:

Can you disclose what hardware you are running this against (i.e. the hardware you are monitoring)?

We are using Huawei 2288H V5 servers.

Also, have you successfully extracted DCMI information from those machines with either ipmi-dcmi or other IPMI tools?

With this HW, I have come up with a workaround:

~$ ipmi-oem IntelNM get-node-manager-statistics mode=globalpower
Current Power                                 : 277 Watts
Minimum Power                                 : 40 Watts
Maximum Power                                 : 665 Watts
Average Power                                 : 403 Watts
Statistics Reporting Period                   : 33855373 seconds
Policy/Global Administrative State            : Disabled
Measurements State                            : In Progress

ipmi-oem is shipped with freeipmi-1.6.8-1.el8.x86_64, on an RHEL 8 compatible Linux distribution. Maybe this information will be useful for other users too, on different hardware platforms.

rezacloner1372 commented 1 year ago

Hi. I have this problem too. my exporter info : ipmi_exporter, version 1.6.1 (branch: HEAD, revision: 344b8b4a565a9ced936aad4d4ac9a29892515cba) build user: root@3cb2b9536d9d build date: 20220617-09:57:08 go version: go1.18.3 platform: linux/amd64

k0ste commented 1 year ago

@rezacloner1372 what exactly issue do you have? What is your server? What is output for ipmi-dcmi --get-system-power-statistics?

rezacloner1372 commented 1 year ago

error : caller=collector_dcmi.go:53 level=error msg="Failed to collect DCMI data" target=[local] error="error running ipmi-dcmi: exit status 1: ipmi_cmd_dcmi_get_power_reading: BMC busy\n

Current Power : Watts Minimum Power over sampling duration : watts Maximum Power over sampling duration : watts Average Power over sampling duration : watts Time Stamp : 06/24/2023 - 03:28:32 Statistics reporting time period : *** milliseconds Power Measurement : Active

*** >>> is the int numbers

k0ste commented 1 year ago

As you can see, something wrong with your BMC or exporter setup

error : caller=collector_dcmi.go:53 level=error msg="Failed to collect DCMI data" target=[local] error="error running ipmi-dcmi: exit status 1: ipmi_cmd_dcmi_get_power_reading: BMC busy\n

"BMC busy" - The BMC is currently busy. It may be processing information or have too many simultaneous sessions to manage. Please wait and try again.

Due exit-code 1 the exporter can't process your metircs

brokenjacobs commented 2 months ago

Also having this issue on Supermicro X13SAZ-F