search

intel / ipmctl

BSD 3-Clause "New" or "Revised" License
184 stars 62 forks source link

AEP firmware downgrade fail #128

Closed Muthu2112 closed 4 years ago

Muthu2112 commented 4 years ago

Firmware downgrade fail using IPMCTL version :01.00.00.3497

I have 4 AEP DIMM and It stuck either pressing y for first dimm or after that

Downgrading firmware from 01.02.00.5435 to 01.02.00.5417

[root@localhost AEP]# ipmctl load -source 5417.bin -dimm Starting update on 4 dimm(s)... Downgrade firmware on DIMM 0x0001? Do you want to continue? [y/n] y Downgrade firmware on DIMM 0x0101? Do you want to continue? [y/n] y

Need your help to resolve

kellycouch commented 4 years ago

Please clarify: after you press 'y' and , there is no further response/output?

Can you run same command with -f option, to skip the prompts?

Muthu2112 commented 4 years ago

Yes no response

tried with ipmctl load -source 5417.bin -f -dimm where same issue happening

nolanhergert commented 4 years ago

Investigating it on our side. Thanks!

nolanhergert commented 4 years ago

I might be able to diagnose your issue as having to do with Address Range Scrub (triggered by Linux after boot) occurring at the same time as firmware update on a particular PMem module, causing the update to take about 90 seconds vs. the normal ~10 seconds. The length of time required for ARS varies based on the size of the PMem module, but for my 512GiB modules it's taking ~5 minutes each. Before running firmware update, I would run ipmctl show -d ARSStatus -dimm to determine if ARS “in progress” is the reason for your observation.

---DimmID=0x0001--- ARSStatus=In progress ---DimmID=0x0011--- ARSStatus=Aborted ---DimmID=0x0101--- ARSStatus=Completed ---DimmID=0x0111--- ARSStatus=Completed

We are able to cancel ARS during firmware update, but due to a bug in our 1.0 code this doesn't occur for firmware update over small payload (which is what runs in OS). This is fixed in 2.0 build 3695, or more specifically this commit: https://github.com/intel/ipmctl/commit/dc0f9f8d5f351033d13ad3754bb08fc57c901ea7

StevenPontsler commented 4 years ago

If someone does not want to upgrade to a newer version of ipmctl can they just wait till that command shows that none of the PMem modules are "In progress" before attempting the firmware update?

Muthu2112 commented 4 years ago

One question we are using 01.00.00.3497 version for all our AEP testing.

  1. Is it correct IPMCTL version to use with AEP firmware 01.02.00.5435 combination ?
  2. How to know which IPMCTL version need to be used for each firmware version series ?
nolanhergert commented 4 years ago

@Muthu2112 I tested with 3497 as well. There should be no limitation on firmware version used, except we don't support future compatibility. (can't use 1. ipmctl with 2. PMem module) We should provide a clear error indicating this situation if you run into it, but it might just surface as an "unmanageable" PMem module.

@StevenPontsler Yes, but I think it'll be faster to just let firmware update run for the 80 extra seconds instead of waiting 5 minutes * N PMem modules.