search

xcat2 / xcat-core

Code repo for xCAT core packages
Eclipse Public License 1.0
367 stars 172 forks source link

Broken communication with BMC on non-default OpenBMC password #6032

Open thilo-maurer opened 5 years ago

thilo-maurer commented 5 years ago

I'm having a compute node with OpenBMC (model IBM 8335-GTH) I'm unable to use xCAT once I change the password away from the default password:

[root@xcat-master ~]# chtab key=openbmc passwd.username=root passwd.password=0penBmc
[root@xcat-master ~]# bmcdiscover -s nmap --range 10.129.0.4
10.129.0.4,8335-GTH,7880C6A,,,mp,bmc,,,

Good. Now changing password via tthe WEB-GUI to abcdefgh.

[root@xcat-master ~]# bmcdiscover -s nmap --range 10.129.0.4
Warning: [xcat-master]: Invalid username or password for 10.129.0.4
[root@xcat-master ~]# chtab key=openbmc passwd.username=root passwd.password=abcdefgh
[root@xcat-master ~]# bmcdiscover -s nmap --range 10.129.0.4
Warning: [xcat-master]: Unable to establish IPMI v2 / RMCP+ session: unauthorized name for 10.129.0.4

Setting back password to default value 0penBmc everything works again.

I see similar issue not only with bmcdiscover, but also for other tools like rpower, etc..

This is with xCAT 2.14.4

xuweibj commented 5 years ago

Hi @thilo-maurer , For bmcdiscover, we just read the default password in our pm file, did not read it from passwd table. Maybe we will modify it later.

But for rpower, I want to make sure something with you. Do you have bmcpassword defined for that node? You could check it in openbmc table or node's definition by lsdef <node>. If defined, we will use it first not passwd.password.

thilo-maurer commented 5 years ago

No, for bmcdiscover it seems the passwd table is read. See log below. Only when the passwd password it set to the default value, then it works.

# Current password set in web-gui of server is "0penBmc"

[root@xcat-master ~]# tabdump passwd
#key,username,password,cryptmethod,authdomain,comments,disable
"openbmc","root","abcdefgh",,,,

[root@xcat-master ~]# bmcdiscover -s nmap --range 10.129.0.4
Warning: [xcat-master]: Unable to establish IPMI v2 / RMCP+ session: unauthorized name for 10.129.0.4

[root@xcat-master ~]# chtab key=openbmc passwd.username=root passwd.password=0penBmc

[root@xcat-master ~]# bmcdiscover -s nmap --range 10.129.0.4
10.129.0.4,8335-GTH,7880C6A,,,mp,bmc,,,

Regarding rpower: I'm taking everything back. I was in error.

cxhong commented 5 years ago

can u try again after remove -s options for bmcdiscover command?

thilo-maurer commented 5 years ago

That gives the same error message

[root@xcat-master ~]# bmcdiscover --range 10.129.0.4
Warning: [xcat-master]: Unable to establish IPMI v2 / RMCP+ session: unauthorized name for 10.129.0.4
thilo-maurer commented 5 years ago

May this helps. When a wrong password is set in the passwd table, the error message is different:

[root@xcat-master ~]# bmcdiscover --range 10.129.0.4
Warning: [xcat-master]: Invalid username or password for 10.129.0.4
cxhong commented 5 years ago

did u able ssh to openbmc with non-default password? also, can u try use ipmitool command from xcat-master?

thilo-maurer commented 5 years ago

SSH works fine.

ipmitool not: 

# ipmitool-xcat -H 10.129.0.4 -U root -P *************** -I lanplus lan print
Error: Unable to establish IPMI v2 / RMCP+ session
cxhong commented 5 years ago

how about ipmitool, not ipmitool-xcat? just want to check if it works without xCAT command.

whowutwut commented 5 years ago

@thilo-maurer Can you show me what version of the FW you are running?

You can do the following and replace <password> with your password

BMCIP=10.129.0.4
curl -c cjar -k -X POST -H 'Content-Type: application/json' -d '{"data": [ "root", "<password>" ] }' https://$BMCIP/login
curl -c cjar -b cjar -k -H 'Content-Type: application/json' -X GET https://$BMCIP/xyz/openbmc_project/software/enumerate
xuweibj commented 5 years ago

@thilo-maurer sorry, you are right, it seems the password abcdefgh was used. Could you help to run this 2 commands after modified password to abcedfgh?

@whowutwut @cxhong I reproduced this in our env. After modified bmc password to abcdefgh by rest, run ipmitool got output:

# ipmitool -I lanplus -H 10.5.14.100 -P abcdefgh mc info
Error: Unable to establish IPMI v2 / RMCP+ session

But 0penBmc still works well:

# ipmitool -I lanplus -H 10.5.14.100 -P 0penBmc mc info
Device ID                 : 0
Device Revision           : 0
Firmware Revision         : 2.00
IPMI Version              : 2.0
Manufacturer ID           : 42817
Manufacturer Name         : Unknown (0xA741)
Product ID                : 16975 (0x424f)
Product Name              : Unknown (0x424F)
Device Available          : yes
Provides Device SDRs      : yes
Additional Device Support :
    Sensor Device
    SEL Device
    FRU Inventory Device
    Chassis Device
Aux Firmware Rev Info     :
    0x00
    0x00
    0x00
    0x46

It seems the password for IPMI interface did not be modified at the same time. Version: ibm-v2.0-0-r46-0-gbed584c

thilo-maurer commented 5 years ago 
[root@xcat-master ~]# curl -c cjar -k -X POST -H 'Content-Type: application/json' -d '{"data": [ "root", "*************" ] }' https://$BMCIP/login                                           
{
  "data": "User 'root' logged in", 
  "message": "200 OK", 
  "status": "ok"
}
[root@xcat-master ~]# curl -c cjar -b cjar -k -H 'Content-Type: application/json' -X GET https://$BMCIP/xyz/openbmc_project/software/enumerate                                                    
{
  "data": {
    "/xyz/openbmc_project/software/376af621": {
      "Activation": "xyz.openbmc_project.Software.Activation.Activations.Active", 
      "Path": "", 
      "Priority": 0, 
      "Purpose": "xyz.openbmc_project.Software.Version.VersionPurpose.BMC", 
      "RequestedActivation": "xyz.openbmc_project.Software.Activation.RequestedActivations.None", 
      "Version": "ibm-v2.1-438-g0030304-r15-0-g19832d3", 
      "associations": [
        [
          "inventory", 
          "activation", 
          "/xyz/openbmc_project/inventory/system/chassis/motherboard/boxelder/bmc"
        ]
      ]
    }, 
    "/xyz/openbmc_project/software/376af621/inventory": {
      "endpoints": [
        "/xyz/openbmc_project/inventory/system/chassis/motherboard/boxelder/bmc"
      ]
    }, 
    "/xyz/openbmc_project/software/376af621/software_version": {
      "endpoints": [
        "/xyz/openbmc_project/software"
      ]
    }, 
    "/xyz/openbmc_project/software/6017da28": {
      "Activation": "xyz.openbmc_project.Software.Activation.Activations.Active", 
      "Path": "", 
      "Priority": 1, 
      "Purpose": "xyz.openbmc_project.Software.Version.VersionPurpose.BMC", 
      "RequestedActivation": "xyz.openbmc_project.Software.Activation.RequestedActivations.None", 
      "Version": "ibm-v2.1-438-g0030304-r12-0-g5ee4fb0", 
      "associations": [
        [
          "inventory", 
          "activation", 
          "/xyz/openbmc_project/inventory/system/chassis/motherboard/boxelder/bmc"
        ]
      ]
    }, 
    "/xyz/openbmc_project/software/6017da28/inventory": {
      "endpoints": [
        "/xyz/openbmc_project/inventory/system/chassis/motherboard/boxelder/bmc"
      ]
    }, 
    "/xyz/openbmc_project/software/6017da28/software_version": {
      "endpoints": [
        "/xyz/openbmc_project/software"
      ]
    }, 
    "/xyz/openbmc_project/software/634fa561": {
      "Activation": "xyz.openbmc_project.Software.Activation.Activations.Active", 
      "ExtendedVersion": "op-build-v2.0.8-1-gc51594f,buildroot-2018.02.1-6-ga8d1126,skiboot-v6.0.7,hostboot-d033213-pfb2e171,occ-084756c,linux-4.16.13-openpower1-p328018f,petitboot-v1.7.2-p8f11e93,machine-xml-7cd20a6,hostboot-binaries-hw080418a.op920,capp-ucode-p9-dd2-v4,sbe-55d6eb2,hcode-hw080418a.op920", 
      "Path": "", 
      "Priority": 0, 
      "Purpose": "xyz.openbmc_project.Software.Version.VersionPurpose.Host", 
      "RequestedActivation": "xyz.openbmc_project.Software.Activation.RequestedActivations.None", 
      "Version": "IBM-witherspoon-ibm-OP9-v2.0.8-2.2-prod", 
      "associations": [
        [
          "inventory", 
          "activation", 
          "/xyz/openbmc_project/inventory/system/chassis"
        ]
      ]
    }, 
    "/xyz/openbmc_project/software/634fa561/inventory": {
      "endpoints": [
        "/xyz/openbmc_project/inventory/system/chassis"
      ]
    }, 
    "/xyz/openbmc_project/software/634fa561/software_version": {
      "endpoints": [
        "/xyz/openbmc_project/software"
      ]
    }, 
    "/xyz/openbmc_project/software/active": {
      "endpoints": [
        "/xyz/openbmc_project/software/634fa561", 
        "/xyz/openbmc_project/software/bc75273f", 
        "/xyz/openbmc_project/software/6017da28", 
        "/xyz/openbmc_project/software/376af621"
      ]
    }, 
    "/xyz/openbmc_project/software/bc75273f": {
      "Activation": "xyz.openbmc_project.Software.Activation.Activations.Active", 
      "ExtendedVersion": "op-build-v2.0-11-gb248194,buildroot-2018.02.1-6-ga8d1126,skiboot-v6.0.1,hostboot-8ab6717d-p110cb65,occ-77bb5e6,linux-4.16.8-openpower2-p6a14c7f,petitboot-v1.7.1-p50a5645,machine-xml-7cd20a6,hostboot-binaries-276bb70,capp-ucode-p9-dd2-v4,sbe-a596975,hcode-b8173e8", 
      "Path": "", 
      "Priority": 255, 
      "Purpose": "xyz.openbmc_project.Software.Version.VersionPurpose.Host", 
      "RequestedActivation": "xyz.openbmc_project.Software.Activation.RequestedActivations.None", 
      "Version": "IBM-witherspoon-ibm-OP9-v2.0-2.14-prod", 
      "associations": [
        [
          "inventory", 
          "activation", 
          "/xyz/openbmc_project/inventory/system/chassis"
        ]
      ]
    }, 
    "/xyz/openbmc_project/software/bc75273f/inventory": {
      "endpoints": [
        "/xyz/openbmc_project/inventory/system/chassis"
      ]
    }, 
    "/xyz/openbmc_project/software/bc75273f/software_version": {
      "endpoints": [
        "/xyz/openbmc_project/software"
      ]
    }, 
    "/xyz/openbmc_project/software/functional": {
      "endpoints": [
        "/xyz/openbmc_project/software/634fa561", 
        "/xyz/openbmc_project/software/376af621"
      ]
    }
  }, 
  "message": "200 OK", 
  "status": "ok"
thilo-maurer commented 5 years ago

Both ipmitool show version 1.8.18

[root@xcat-master ~]# ipmitool -I lanplus -H 10.129.0.4 -P abcedfgh mc info                                                                                                                       
Error: Unable to establish IPMI v2 / RMCP+ session
[root@xcat-master ~]# ipmitool-xcat -I lanplus -H 10.129.0.4 -P abcedfgh mc info                                                                                                                  
Error: Unable to establish IPMI v2 / RMCP+ session
xuweibj commented 5 years ago

https://github.com/openbmc/openbmc/issues/3487

robin2008 commented 5 years ago

Except ipmitool, do we have other method to get mandatory information to finish discovering (for example, REST or ssh)?

xuweibj commented 5 years ago

Based on issue openbmc/openbmc#3487, seems that for older BMC version could not modify password of ipmi interface. So we need to use default password 0penBmc to connect BMC to get information. And for new version we need to modify code to support it. Will discuss how to solve this issue and support new BMC version.

thilo-maurer commented 5 years ago

Upgraded to latest Firmware. IPMI password remains unchanged. See https://github.com/openbmc/openbmc/issues/3487

I know xCAT relies on impitool to do the communication with the BMCs. For IBM servers with OpenBMC, IBM provides the tool openbmctool at http://www14.software.ibm.com/webapp/set2/sas/f/lopdiags/scaleOutLCdebugtool.html#OpenBMC and this one seems to not have the issue above.

Is this one used by xCAT, or is ipmitool used exclusively?

xuweibj commented 5 years ago

@thilo-maurer Actually, to handle BMC and OpenBMC these 2 types bmcdiscover, our logic is to get information by ipmitool first to judge whether it is OpenBMC or BMC. We must make sure it is OpenBMC and then run command like openbmctool to get more information. If not do this, run command like openbmctool against that node directly will get strange message it the node is not BMC or OpenBMC. So we need to discuss how to check the node type again for this scenario.

robin2008 commented 5 years ago

@xuweibj @zet809 Maybe we need a more smart method to determine if the node model.

thilo-maurer commented 5 years ago

I don't see a reason to really change xCAT at this point. OpenBMC should simply configure their IPMI server to allow for access using IPMItool with proper credentials. What specific ipmi command are using?

Is it ipmitool fru list?

xuweibj commented 5 years ago

ipmitool -I lanplus -H <bmcip> -P <password> mc info To get information: Manufacturer ID and Product ID.