FVT1060:Everest:SEL clear command not clearing the entries via IPMI

yadlapati commented 9 months ago

Steps to re- create

======================

Login to Rainier system Executing the IPMI command to clear the entries:

bash-4.2$ ipmitool -I lanplus -C 17 -N 3 -p 623 -U ipmi_admin -H rain57bmc.aus.stglabs.ibm.com sel list 2>&1

Password:

1 | 02/20/2024 | 05:22:19 | System Event #0x4a | Undetermined system hardware failure | Asserted

bash-4.2$ ipmitool -I lanplus -C 17 -N 3 -p 623 -U ipmi_admin -H rain57bmc.aus.stglabs.ibm.com sel clear 2>&1

Password:

Clearing SEL. Please allow a few seconds to erase.

bash-4.2$ ipmitool -I lanplus -C 17 -N 3 -p 623 -U ipmi_admin -H rain57bmc.aus.stglabs.ibm.com sel list 2>&1

Password:

1 | 02/20/2024 | 06:02:42 | System Event #0x4a | Undetermined system hardware failure | Asserted

Actual behaviour: IPMI sel clear command is not clearing the entries

Expected Behaviour: IPMI command need to clear and display has SEL has no entries.

lxwinspur commented 9 months ago

@yadlapati I have a couple of questions:

Why are the times before and after executing sel clear different?

I wonder if the second sel is newly generated?

When you execute the following command, can the following log be displayed?


bash-4.2$ ipmitool -I lanplus -C 17 -N 3 -p 623 -U ipmi_admin -H rain57bmc.aus.stglabs.ibm.com sel clear 2>&1

Password:

Clearing SEL. Please allow a few seconds to erase.


**show journalctl log**
`journalctl -b | grep ipmi`

mzipse commented 8 months ago

@lxwinspur , passing this info along to you from our tester....

The below is newly capture data with no much time difference :

For reference below is the journal logs.

root@p10bmc:~# journalctl -b | grep ipmi Mar 06 07:00:28 p10bmc kernel: ipmi-bt-host 1e789140.ibt: Found bt bmc device Mar 06 07:00:28 p10bmc kernel: ipmi-bt-host 1e789140.ibt: Using IRQ 33 Mar 06 07:00:29 p10bmc systemd[1]: /usr/lib/systemd/system/phosphor-ipmi-net@.socket:6: Invalid interface name, ignoring: sys-subsystem-net-devices-%i.device Mar 06 07:00:29 p10bmc systemd[1]: /usr/lib/systemd/system/phosphor-ipmi-net@.socket:6: Invalid interface name, ignoring: sys-subsystem-net-devices-%i.device Mar 06 07:00:29 p10bmc systemd[1]: Created slice Slice /system/phosphor-ipmi-net. Mar 06 07:00:38 p10bmc systemd[1]: Listening on phosphor-ipmi-net@eth0.socket. Mar 06 07:00:38 p10bmc systemd[1]: Listening on phosphor-ipmi-net@eth1.socket. Mar 06 07:00:53 p10bmc ipmid[573]: JSON file not found Mar 06 07:00:54 p10bmc ipmid[573]: Loading whitelist filter Mar 06 07:00:55 p10bmc ipmid[573]: Set restrictedMode = false Mar 06 07:00:56 p10bmc systemd[1]: First Boot Disable IPMI Network was skipped because of an unmet condition check (ConditionPathExists=!/etc/ipmi-net-disable-one-time). Mar 06 07:00:57 p10bmc netipmid[805]: Failed to get bus name, path: /org/openbmc/control/chassis0, error: Input/output error Mar 06 07:00:57 p10bmc netipmid[805]: Bind to interface: eth0 Mar 06 07:00:57 p10bmc ipmid[573]: New interface mapping Mar 06 07:00:57 p10bmc netipmid[807]: Failed to get bus name, path: /org/openbmc/control/chassis0, error: Input/output error Mar 06 07:00:57 p10bmc netipmid[807]: Bind to interface: eth1 Mar 06 07:00:57 p10bmc ipmid[573]: New interface mapping Mar 06 07:01:10 p10bmc ipmid[573]: Host control timeout hit! Mar 06 07:01:10 p10bmc ipmid[573]: Failed to deliver host command Mar 06 07:02:46 p10bmc useradd[2517]: new user: name=ipmi_admin, UID=1002, GID=100, home=/home/ipmi_admin, shell=/sbin/nologin, from=none Mar 06 07:02:46 p10bmc useradd[2517]: add 'ipmi_admin' to group 'priv-admin' Mar 06 07:02:46 p10bmc useradd[2517]: add 'ipmi_admin' to group 'web' Mar 06 07:02:46 p10bmc useradd[2517]: add 'ipmi_admin' to group 'redfish' Mar 06 07:02:46 p10bmc useradd[2517]: add 'ipmi_admin' to shadow group 'priv-admin' Mar 06 07:02:46 p10bmc useradd[2517]: add 'ipmi_admin' to shadow group 'web' Mar 06 07:02:46 p10bmc useradd[2517]: add 'ipmi_admin' to shadow group 'redfish' Mar 06 07:02:46 p10bmc phosphor-user-manager[2517]: useradd: warning: the home directory /home/ipmi_admin already exists. Mar 06 07:02:46 p10bmc bmcweb[1808]: pam_unix(webserver:chauthtok): password changed for ipmi_admin Mar 06 07:04:08 p10bmc usermod[2569]: delete 'ipmi_admin' from group 'web' Mar 06 07:04:08 p10bmc usermod[2569]: delete 'ipmi_admin' from group 'redfish' Mar 06 07:04:08 p10bmc usermod[2569]: add 'ipmi_admin' to group 'ipmi' Mar 06 07:04:08 p10bmc usermod[2569]: delete 'ipmi_admin' from shadow group 'web' Mar 06 07:04:08 p10bmc usermod[2569]: delete 'ipmi_admin' from shadow group 'redfish' Mar 06 07:04:08 p10bmc usermod[2569]: add 'ipmi_admin' to shadow group 'ipmi' Mar 06 07:05:16 p10bmc ipmid[573]: Failed to fetch service for D-Bus object Mar 06 07:05:16 p10bmc netipmid[805]: Removing idle IPMI LAN session, id: 857582077, handler: 1 Mar 06 07:17:32 p10bmc ipmid[573]: Command in process, no attention Mar 06 07:17:35 p10bmc ipmid[573]: Host control timeout hit! Mar 06 07:17:35 p10bmc ipmid[573]: Failed to deliver host command Mar 06 07:17:35 p10bmc ipmid[573]: Failed to deliver host command Mar 06 07:20:00 p10bmc netipmid[805]: Removing idle IPMI LAN session, id: 866293172, handler: 1 Mar 06 07:20:18 p10bmc netipmid[805]: Removing idle IPMI LAN session, id: 1041173208, handler: 1 Mar 06 07:21:10 p10bmc netipmid[805]: Removing idle IPMI LAN session, id: 914219419, handler: 1 Mar 06 07:21:15 p10bmc netipmid[805]: Removing idle IPMI LAN session, id: 908241199, handler: 1 Mar 06 07:23:37 p10bmc netipmid[805]: Removing idle IPMI LAN session, id: 171392366, handler: 1

lxwinspur commented 8 months ago

bash-4.2$ ipmitool -I lanplus -C 17 -N 3 -p 623 -U ipmi_admin -H rain57bmc.aus.stglabs.ibm.com sel list 2>&1 Password: 1 | 03/06/2024 | 07:20:18 | System Event #0x4a | Undetermined system hardware failure | Asserted 2 | 03/06/2024 | 07:20:33 | Unknown #0x49 | State Deasserted | Asserted

I think this is correct. After sel clear is executed, all sel data has been deleted, and these two pieces of data are newly generated (please pay attention to the generation time)

yadlapati commented 8 months ago

@@lxwinspur I think that is correct. Those 2 are the newer SELs

lxwinspur commented 8 months ago

so can we close this issue?

mzipse commented 7 months ago

@lxwinspur , our test team is still concerned with this and quite honestly, it seems odd to me as well. Why would we surface an error when deleting the error log? "Undetermine system hardware failure" doesn't seem like the correct message to our customers. I also got this feedback from our test team.....

"I attempted clear all SEL and then generate an unrecoverable error log in eBMC. Then, I tried to look up details in the IPMI SEL(See below). Here, I noticed that the SEL entry for both error logs is the same. This could confuse the customer. Therefore, we should correct the description of SEL logs that are logged for clearing SEL.

Shouldn't there be some differentiation between the SEL for deleting the error log vs a SEL for a real hardware failure?

I realize this is lower priority for the IPS team right now. As time permits, please investigate and let me know what you think.

lxwinspur commented 7 months ago

sel clear will only clear the previously generated sel logs. It does not care about the log type and log content, so the two newly generated logs are not what sel clear is concerned about. We should see under what circumstances these two logs are generated, but I think they should be two issues right?

mzipse commented 1 month ago

This was tracked internally as 598769 and we closed it has a permanent restriction primarily because it's not something we expect our customers to do. So I'm going to close this github issue as well.

ibm-openbmc / dev

FVT1060:Everest:SEL clear command not clearing the entries via IPMI #3638