What does "Maintenance UE" mean

[Grubby0624]

https://github.com/open-power/hostboot/blob/9c7a1381407efe2d5dcabbfadeda0f8639c34c95/src/usr/diag/prdf/common/plat/mem/prdfMemExtraSig.H#L43

When we bring up on the Rainier platform, the following error was reported in istep14.1: "Maintenance AUE" and "Maintenance UE" Excuse me? What do the two errors mean respectively? Is there a suggested debug direction

[mabaiocchi]

I'm going to yield to @zane131 or @cnpalmer to better answer this question.

[cnpalmer]

Address uncorrectable errors (AUEs) and uncorrectable errors (UEs) detected on a maintenance read of memory. I believe both will callout the rank of memory the error was found on which would probably end up as a dimm replacement. The AUE also calls out the port at lower priority. The logs should specify what hardware was called out.

[zane131]

If this is on the initial bring up, you could try checking if the memory DIMMs are seated properly in their slots. Otherwise, replacement may be necessary.

[Grubby0624]

When I reduce the memory frequency to 2666MHz, this error will not be reported. I think this may be a signal integrity problem.

[Grubby0624]

Excuse me?

1. Are there any suggestions for testing and improving the signal integrity of this DIMM
2. I remember how P9 can improve signal integrity by directly modifying registers when debugging LRDIMMs. Is there a corresponding method in P10: https://github.com/open-power/hostboot/blob/859507a61453b80b119d8da8bf28654f5799f896/src/import/chips/p9/procedures/hwp/memory/lib/phy/mss_lrdimm_training_helper.C#L802

[dcrowell77]

@esteban012 might be able to help. There are many many knobs to turn on both the P10 MC and also the Explorer logic but I definitely don't know what they are personally.

[sglancy6]

All current workarounds are included within the latest firmware available on the OpenPOWER github. If you think that the error is caused by a signal integrity issue, then I would strongly recommend running shmoo tests to verify that the margins are sufficient.

[Grubby0624]

Thanks for your reply. Our shmoo test on Rainier is currently in the process of trial. There are two more questions to confirm:

1. An error is reported at istep 14.1 at 3200MHz. DDR training has run before, and all LANEs returned to us by our print explorer are PASSING. Is it still possible that the problem is signal integrity

2. What is the specific process of MCBIST? Is it a bist to MC or a bist to DRAM? Is it the same thing as the mbist in DDR?

[sglancy6]

1. It's possible. Training runs on a limited subset of the addresses.
2. MCBIST is run on the memory controller and runs traffic over to the DRAM. It is not the same as mbist on the DDR.

[Grubby0624]

We find that the problem is due to the value of the following registers that the PRDF locates:

EXPLR RDF FIR=0x8011C00 (its value is 0x000000809e0000000). I understand that this is the RDF FIR register of Explorer. Is there any more information available? Would you please explain more on "RDF"? In addition, I have a limited number of DDIMMs, so I want to try to repair this DDIMM through firmware configuration, based on above, would you please help advice on below questions? Thank you!

1. Is there any possibility of firmware repair for this error

2. If so, could you give me some suggestions for debugging

[sglancy6]

I'm seeing that the value sent above is 68 bits of data: 0x000000809e0000000

Which of the following is correct:

1. 0x00000809e0000000
2. 0x000000809e000000

[Grubby0624]

This is the correct value: 0x00000809e0000000

[sglancy6]

The RDF_FIR register is reporting errors found during the maintenance commands:

mark place error on rank 0 new correctable error detected on a maintenance read chip mark corrected error on a maintenance read

special uncorrectable error on a maintenance read address uncorrectable error on a maintenance read uncorrectable error on a maintenance read

My recommendation is to run MCBIST and shmoo to see if a simpler test case fails to do memory writes and reads.

[liuxiwei1013]

This question comes from @Grubby0624: I think I have resolved the error in "Maintenance UE", which was caused by the Address signal quality not meeting the standard. After adjusting the value of "ATxDly-A/B", it can be turned on, but after running HTX for a long time, it still reports an MCE error. I ran the shmoo test on ATX0 and found that its result is smaller than other DDIMM values: 2024-01-03 T09:32:46.913845697, [TRC]: Minimum margins: explorer: k0: n0: s0: p02 Minimum Setup PS: 107, Minimum Hold PS: 49, Minimum Eye PS: 156 I would like to ask for more advice: 1) What is the standard for ATX0 test pass? What are the minimum setup/hold times 2) I think the Eye of this memory is not up to standard. If I want to increase this value, which parameter should I adjust?

[ecorderoibm]

@liuxiwei1013 is that schmo result the same explorer reporting the CE's ? How fast are you running the interface?

[Grubby0624]

1. The system can boot up to the OS with only this one DDIMM, but running HTX for a long time will report MCE, HTX will terminate, but the system will not crash
2. "How fast are you running the interface?" Are you referring to the rate of DDR? 3200MHz

[ecorderoibm]

You are using industry standard dimms correct? Does the MCE get reported when it crashes or is it already there during the long run you describe? I don't see how the MCE could cause the HTX to stop. I am still looking to get the margins expected from the schmoo. The eye seems small, but I want to make sure. What did you do to fix the address bus signal issue? Could that still be borderline? Thank you @Grubby0624

[Grubby0624]

1. MCE get reported when it crash，According to my understanding, if MCE occurs while running HTX mdt.mem, HTX should stop.
2. I modified the parameter values sent to OCMB in the PHYINIT command, based on Shmoo testing, show that the eye width of Address/Cmd is relatively small. I have modified the following parameters: AtxImpedance atxSlewRate ATxDly A/B [0-7] I'm not sure if it's a critical point or if there are other reasons that are causing MCE errors now

[Grubby0624]

I tried to adjust the values of the following parameters: atxImpedance/atxSlewRate/cktxImpedance, and finally found the best ATX-0 eye width of 254ps. Afterwards, we made many modifications to the SI parameters, but the results did not improve. May I ask:

1. What was done specifically for this ATX test? What signal is it based on and what is the result of sampling?

2. What may be the reason for the small eye size of ATX, and what methods do we have to improve this result.

Thank you!