search

openbmc / phosphor-host-ipmid

dbus-based ipmid for host-endpoint IPMI commands
Apache License 2.0
37 stars 75 forks source link

ipmitool lan print command returns wrong IP Adress #207

Open chnguyen-ampere opened 2 months ago

chnguyen-ampere commented 2 months ago

Hi all, I'm testing on master branch and at e7ef94d350cd156c54a5789ce7d53eb1a55f7da9 (Commits on Aug 30, 2024 ).

Issue: The ipmitool lan print command returns wrong IP Address after I switch IP Address Source from Static to DHCP

Step to reproduce:

  1. Wait BMC boot completely
  2. Check the IP Address Source and IP Address by ipmitool lan print command 
    # ipmitool lan print 1
IP Address Source       : DHCP Address
IP Address              : 10.38.153.72
  3. Switch to Static source by command 
    # ipmitool lan set 1 ipsrc static
  4. Set IP Address to a static address (for example 10.38.153.70) 
    # ipmitool lan set 1 ipaddr 10.38.153.70
Setting LAN IP Address to 10.38.153.70
  5. Check the IP Address Source and IP Address by ipmitool lan print command 
    # ipmitool lan print 1
IP Address Source       : Static Address
IP Address              : 10.38.153.70
  6. Switch back DHCP source by command 
    # ipmitool lan set 1 ipsrc dhcp
  7. Check the IP Address Source and IP Address by ipmitool lan print command 
    # ipmitool lan print 1
IP Address Source       : DHCP Address
IP Address              : 10.38.153.70

    networkctl status on the eth0

    * 2: eth0
               Link File: /usr/lib/systemd/network/99-default.link
            Network File: /etc/systemd/network/00-bmc-eth0.network
                          `-/run/systemd/network/00-bmc-eth0.network.d/updated.conf
                   State: routable (configured)
            Online state: online
                    Type: ether
                    Path: platform-1e660000.ethernet
                  Driver: ftgmac100
        Hardware Address: 9c:c2:c4:53:7d:77
                     MTU: 1500 (min: 68, max: 1500)
                   QDisc: pfifo_fast
IPv6 Address Generation Mode: eui64
Number of Queues (Tx/Rx): 1/1
        Auto negotiation: yes
                   Speed: 1Gbps
                  Duplex: full
                    Port: tp
                 Address: 10.38.153.70
                          10.38.153.72 (DHCP4 via 10.38.152.2)
                          fe80::9ec2:c4ff:fe53:7d77
                 Gateway: 10.38.153.1
                          fe80::7626:acff:fe8d:6c81
                          fe80::82e0:1dff:fe73:2cfc
                          fe80::82e0:1dff:fef4:541
                     DNS: 10.38.18.70
                          10.38.18.71
          Search Domains: amperecomputing.com
       Activation Policy: up
     Required For Online: yes
         DHCP4 Client ID: 9c:c2:c4:53:7d:77
       DHCP6 Client DUID: DUID-EN/Vendor:0000ab11c0a4f695bfe91435
            Connected To: switch on port Ethernet1/3 (Ethernet1/3)

    Dbus object on the 10.38.153.70

    root@mtmitchell-dcscm:~# busctl introspect xyz.openbmc_project.Network /xyz/openbmc_project/network/eth0/_310_2e38_2e153_2e70_2f32
NAME                                TYPE      SIGNATURE RESULT/VALUE                             FLAGS
org.freedesktop.DBus.Introspectable interface -         -                                        -
.Introspect                         method    -         s                                        -
org.freedesktop.DBus.Peer           interface -         -                                        -
.GetMachineId                       method    -         s                                        -
.Ping                               method    -         -                                        -
org.freedesktop.DBus.Properties     interface -         -                                        -
.Get                                method    ss        v                                        -
.GetAll                             method    s         a{sv}                                    -
.Set                                method    ssv       -                                        -
.PropertiesChanged                  signal    sa{sv}as  -                                        -
xyz.openbmc_project.Network.IP      interface -         -                                        -
.Address                            property  s         "10.38.153.70"                           emits-change writable
.Gateway                            property  s         ""                                       emits-change writable
.Origin                             property  s         "xyz.openbmc_project.Network.IP.Addre... emits-change writable
.PrefixLength                       property  y         32                                       emits-change writable
.Type                               property  s         "xyz.openbmc_project.Network.IP.Proto... emits-change writable
xyz.openbmc_project.Object.Delete   interface -         -                                        -
.Delete                             method    -         -                                        -
# busctl get-property xyz.openbmc_project.Network /xyz/openbmc_project/network/eth0/_310_2e38_2e153_2e70_2f32 xyz.openbmc_project.Network.IP Origin
s "xyz.openbmc_project.Network.IP.AddressOrigin.Static"

Dbus object on the 10.38.153.72

# busctl introspect xyz.openbmc_project.Network /xyz/openbmc_project/network/eth0/_310_2e38_2e153_2e72_2f24
NAME                                TYPE      SIGNATURE RESULT/VALUE                             FLAGS
org.freedesktop.DBus.Introspectable interface -         -                                        -
.Introspect                         method    -         s                                        -
org.freedesktop.DBus.Peer           interface -         -                                        -
.GetMachineId                       method    -         s                                        -
.Ping                               method    -         -                                        -
org.freedesktop.DBus.Properties     interface -         -                                        -
.Get                                method    ss        v                                        -
.GetAll                             method    s         a{sv}                                    -
.Set                                method    ssv       -                                        -
.PropertiesChanged                  signal    sa{sv}as  -                                        -
xyz.openbmc_project.Network.IP      interface -         -                                        -
.Address                            property  s         "10.38.153.72"                           emits-change writable
.Gateway                            property  s         ""                                       emits-change writable
.Origin                             property  s         "xyz.openbmc_project.Network.IP.Addre... emits-change writable
.PrefixLength                       property  y         24                                       emits-change writable
.Type                               property  s         "xyz.openbmc_project.Network.IP.Proto... emits-change writable
xyz.openbmc_project.Object.Delete   interface -         -                                        -
.Delete                             method    -         -                                        -
# busctl get-property xyz.openbmc_project.Network /xyz/openbmc_project/network/eth0/_310_2e38_2e153_2e72_2f24 xyz.openbmc_project.Network.IP  Origin
s "xyz.openbmc_project.Network.IP.AddressOrigin.DHCP"

Expectation: The step#8 returns correct the IP Address Source (DHCP Address) and IP Address (10.38.153.72) (as mentioned at step#2) Actual: The step#8 returns wrong IP Address (10.38.153.70) for DHCP source. The 10.38.153.70 is Static source (as mentioned at step#6)

williamspatrick commented 2 months ago

Do you have the networkctl status output at step 8? How about the dbus tree from phosphor-networkd?

chnguyen-ampere commented 2 months ago

Do you have the networkctl status output at step 8? How about the dbus tree from phosphor-networkd?

Hi @williamspatrick , I updated the log for step#8, you can review that. I saw the netwrorkctl and dbus object report correct value 10.38.153.72 => DHCP ; 10.38.153.70 => Static

williamspatrick commented 2 months ago

It looks like both IP addresses are enabled and it seems like the IPMI code is just picking one of them?

chnguyen-ampere commented 2 months ago

It looks like both IP addresses are enabled and it seems like the IPMI code is just picking one of them?

If the IPMI code is just picking one them then it should be 10.38.153.72 DHCP or 10.38.153.70 Static. Actually, the IPMI reports wrong values 10.38.153.70 DHCP (step#8 shows that)

mdmillerii commented 2 months ago

Phosphor networkd is checking if the interface is DHCP or static and assigning address found on the interface accordingly https://github.com/openbmc/phosphor-networkd/blob/5a45606103157291747f95efc77ba989fa43c97f/src/ethernet_interface.cpp#L161

Ipmi host is not considering more than one address could exist.

Not sure how long systemd networkd has offered the dynamic source or if the address is tagged to netlink.

Why wasn't the static address removed when DHCP was enabled? Is this related to the recent "issue" that when DHCP was enabled without a reachable DHCP server it became unreachable?

By design systemd networkd won't remove an address assigned when it started

thangqn-ampere commented 1 month ago

Look like IPMI always gets the first IP address from ip a, without checking the IP Address Source and the IP Address corresponding to the mode. With below output from ip a,

# ip a show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 9c:c2:c4:53:7c:90 brd ff:ff:ff:ff:ff:ff
    inet 10.77.108.196/24 brd 10.77.108.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.77.108.96/24 brd 10.77.108.255 scope global secondary dynamic eth0
       valid_lft 604770sec preferred_lft 604770sec
    inet6 fe80::9ec2:c4ff:fe53:7c90/64 scope link
       valid_lft forever preferred_lft forever

IPMI will return IP Addresss 10.77.108.196.

mdmillerii commented 1 month ago

As I said it's looking at the address independently of the DHCP status.

The real question is why does the static address still exist after step 7 changing back to DHCP

thangqn-ampere commented 1 month ago

As I know, per https://github.com/openbmc/phosphor-networkd/blob/master/docs/Network-Configuration.md#ipaddress-objects, phosphor-networkd supports multiple IP addresses for an Ethernet interface. And it supports at least a static IP and a dynamic IP at the same time, reporting to Redfish EthernetInterface schema. The issue just comes when working with IPMI that just supports 1 IP per an interface. As the phosphor-networkd has information, IPMI should check from dbus to report correct information

mdmillerii commented 1 month ago

I don't see anything in your reference about DHCP vs static address.

I haven't checked redfish but the reference I made above shows getting the DHCP vs static assignment from the interface object independent of the addresses. (I'm replying from my phone but traced the path last week). I didn't think both kinds simultaneously in one interface was supported but could be remembered during transition to ease transition during PATCH

thangqn-ampere commented 1 month ago

The link I posted indicates that multiple IP addresses is accepted. And I believe that phosphor-networkd is the background application for Ethernet feature. Let me show my test on IPMI commands:

chnguyen-ampere commented 1 week ago

It looks like both IP addresses are enabled and it seems like the IPMI code is just picking one of them?

Yes @williamspatrick , In case both IP address assigned the IPMI code just return one of them. More detail, the IPMI code just return the first object dbus in the eth0 interface. The phosphor-host-ipmid fixed the index that always is 0 https://github.com/openbmc/phosphor-host-ipmid/blob/master/transporthandler.cpp#L280

chnguyen-ampere commented 1 week ago

As I said it's looking at the address independently of the DHCP status.

The real question is why does the static address still exist after step 7 changing back to DHCP

Yes @mdmillerii, Sorry for late reply! I spend more time to debug on the phosphor-host-ipmid and phosphor-networkd.

Agree @mdmillerii , it's looking at the IP address and DHCP status independently.

chnguyen-ampere commented 1 week ago

As I said it's looking at the address independently of the DHCP status.

The real question is why does the static address still exist after step 7 changing back to DHCP

After changing to DHCP mode, then the static address still exists. Because actually the DHCP switching command (as step 7 mentioned ipmitool lan set 1 ipsrc dhcp), that is a DHCP enable dbus command; it calls to phosphor-networkd only for enable DHCP mode (refer this dbus command here https://github.com/openbmc/phosphor-networkd/blob/master/docs/Network-Configuration.md#configure-dhcp). It doesn't delete any other IP address. 

In order to delete the static address, we need to use a other command (https://github.com/openbmc/phosphor-networkd/blob/master/docs/Network-Configuration.md#delete-ip-address)

chnguyen-ampere commented 1 week ago

Hi @williamspatrick and @mdmillerii , I think we should implement a simultaneous between IP Address and IP Address source instead of the hard code for index object lookup always being 0 (as mentioned at https://github.com/openbmc/phosphor-host-ipmid/blob/master/transporthandler.cpp#L280)

williamspatrick commented 1 week ago

implement a simultaneous between IP Address and IP Address source

I don't know what that means.

It seems fine to me if you want to propose a change to the ipmi code to handle this better. The intuitive thing to me would be to prioritize the addresses (and types) so that DHCP is prioritized over static, if it is available. There is probably nothing that is 100% right in all scenarios from all people's perspectives, but that is a limitation of IPMI.

chnguyen-ampere commented 1 week ago

It seems fine to me if you want to propose a change to the ipmi code to handle this better. The intuitive thing to me would be to prioritize the addresses (and types) so that DHCP is prioritized over static, if it is available. There is probably nothing that is 100% right in all scenarios from all people's perspectives, but that is a limitation of IPMI.

Thank @williamspatrick , I'm preparing the patch. I'll push to gerrit soon and add you as a reviewer.