search

ibm-openbmc / dev

Product Development Project Mgmt and Tracking
16 stars 2 forks source link

BMC fails to process remote ipmitool commands consistently #3641

Closed yadlapati closed 2 weeks ago

yadlapati commented 2 months ago

This problem is seen internally on the BMC of our lab

Firmware level: op940.hmc-29-0-g11d2cfce07 IBM-mowgli-ibm-OP9_v2.5_4.127-prod

PROBLEM:

When sending remote ipmitool commands to the BMC, it is able to respond correctly to the first few sent (sometimes many), but then it stops responding and produces error: Error: Unable to establish IPMI v2 / RMCP+ session

Here is an example of an instance where it works:

❯ ipmitool -I lanplus -H pe241.aus.stglabs.ibm.com -U root mc info Password: Unable to Get Channel Cipher Suites

Error: no response from RAKP 1 message Error: Received an Unexpected RAKP 2 message Device ID                 : 0 Device Revision           : 0 Firmware Revision         : 9.40 IPMI Version              : 2.0 Manufacturer ID           : 42817 Manufacturer Name         : IBM Platform Firmware Division Product ID                : 2 (0x0002) Product Name              : Unknown (0x02) Device Available          : yes Provides Device SDRs      : yes Additional Device Support :     Sensor Device     SEL Device     FRU Inventory Device     Chassis Device Aux Firmware Rev Info     :     0x14     0x0b     0x00     0x4f

We have tried including the "-C 17" flag, and the results are similar, except for the suppression of the "Unable to Get Cipher Suites" portion.

If the BMC is rebooted, then it responds to commands again, for a time, and eventually stops responding.

Here is one run with -vvv:

❯ ipmitool -vvv -I lanplus -H pe241.aus.stglabs.ibm.com -U root -P 0penBmc0 mc info ipmitool version 1.8.19

Loading IANA PEN Registry...

Sending IPMI command payload     netfn   : 0x06     command : 0x38     data    : 0x8e 0x04

BUILDING A v1.5 COMMAND

IPMI Request Session Header    Authtype   : NONE    Sequence   : 0x00000000    Session ID : 0x00000000 IPMI Request Message Header    Rs Addr    : 20    NetFn      : 06    Rs LUN     : 0    Rq Addr    : 81    Rq Seq     : 00    Rq Lun     : 0    Command    : 38

Sending IPMI command payload     netfn   : 0x06     command : 0x38     data    : 0x8e 0x04

BUILDING A v1.5 COMMAND

IPMI Request Session Header    Authtype   : NONE    Sequence   : 0x00000000    Session ID : 0x00000000 IPMI Request Message Header    Rs Addr    : 20    NetFn      : 06    Rs LUN     : 0    Rq Addr    : 81    Rq Seq     : 00    Rq Lun     : 0    Command    : 38

Sending IPMI command payload     netfn   : 0x06     command : 0x38     data    : 0x8e 0x04

BUILDING A v1.5 COMMAND

IPMI Request Session Header    Authtype   : NONE    Sequence   : 0x00000000    Session ID : 0x00000000 IPMI Request Message Header    Rs Addr    : 20    NetFn      : 06    Rs LUN     : 0    Rq Addr    : 81    Rq Seq     : 00    Rq Lun     : 0    Command    : 38

Sending IPMI command payload     netfn   : 0x06     command : 0x38     data    : 0x8e 0x04

BUILDING A v1.5 COMMAND

IPMI Request Session Header    Authtype   : NONE    Sequence   : 0x00000000    Session ID : 0x00000000 IPMI Request Message Header    Rs Addr    : 20    NetFn      : 06    Rs LUN     : 0    Rq Addr    : 81    Rq Seq     : 00    Rq Lun     : 0    Command    : 38

Sending IPMI command payload     netfn   : 0x06     command : 0x38     data    : 0x0e 0x04

BUILDING A v1.5 COMMAND

IPMI Request Session Header    Authtype   : NONE    Sequence   : 0x00000000    Session ID : 0x00000000 IPMI Request Message Header    Rs Addr    : 20    NetFn      : 06    Rs LUN     : 0    Rq Addr    : 81    Rq Seq     : 00    Rq Lun     : 0    Command    : 38

Sending IPMI command payload     netfn   : 0x06     command : 0x38     data    : 0x0e 0x04

BUILDING A v1.5 COMMAND

IPMI Request Session Header    Authtype   : NONE    Sequence   : 0x00000000    Session ID : 0x00000000 IPMI Request Message Header    Rs Addr    : 20    NetFn      : 06    Rs LUN     : 0    Rq Addr    : 81    Rq Seq     : 00    Rq Lun     : 0    Command    : 38

Sending IPMI command payload     netfn   : 0x06     command : 0x38     data    : 0x0e 0x04

BUILDING A v1.5 COMMAND

IPMI Request Session Header    Authtype   : NONE    Sequence   : 0x00000000    Session ID : 0x00000000 IPMI Request Message Header    Rs Addr    : 20    NetFn      : 06    Rs LUN     : 0    Rq Addr    : 81    Rq Seq     : 00    Rq Lun     : 0    Command    : 38

Sending IPMI command payload     netfn   : 0x06     command : 0x38     data    : 0x0e 0x04

BUILDING A v1.5 COMMAND

IPMI Request Session Header    Authtype   : NONE    Sequence   : 0x00000000    Session ID : 0x00000000 IPMI Request Message Header    Rs Addr    : 20    NetFn      : 06    Rs LUN     : 0    Rq Addr    : 81    Rq Seq     : 00    Rq Lun     : 0    Command    : 38 Get Auth Capabilities error Error issuing Get Channel Authentication Capabilities request Error: Unable to establish IPMI v2 / RMCP+ session

In one instance it stopped responding after 5 or so commands, in another it took 30 plus.

yadlapati commented 2 months ago

Internal EWM Defect 609396

yadlapati commented 2 weeks ago

This issue was seen only when ipmitool executed from Macbook. This was not a client issue. I am closing this issues as limitation.