search

open-power / hostboot

System initialization firmware for Power systems
Apache License 2.0
74 stars 97 forks source link

Is the memory stream performance related to Signal integrity #228

Closed Grubby0624 closed 11 months ago

Grubby0624 commented 1 year ago

https://github.com/open-power/hostboot/blob/abc40c539e2cb3cb5d7c1473ad9122488b3a595c/src/import/chips/ocmb/explorer/procedures/hwp/memory/exp_draminit.C#L62 During the stream test, I found that the measured stream result is less than 50% of the theoretical value. I think that Signal integrity only affects whether memory initialization can be successful. May I ask for your advice: Does Signal integrity affect memory stream performance? If I want to debug this problem, do I need to pay attention to the Signal integrity parameters I choose?

dcrowell77 commented 1 year ago

We aren't sure what you mean by "Signal integrity" here. If you are talking about electrical issues on any of the busses that can definitely impact performance. However, you mentioned "Signal integrity parameters" which makes me think you are referring to some kind of setting. Can you provide more detail on what/where you are using these parameters?

Also, where are you measuring your results and how? OCMB to DDR? Proc to OCMB? The complete path?

ecorderoibm commented 1 year ago

I don't suspect that the signal integrity settings (assuming you're talking about DDR) would cause an impact to the performance. It should, like you say, really get you to be able to initialize. If you have some settings you might suspect, we can take look at those. I would think you would want to look at the MC settings first though. I suspect it be more likely in there.

Grubby0624 commented 1 year ago

  1. On our platform, the memory Training cannot pass, so I modified some parameters in the' user_input_msdg 'sent by PNOR to OCMB to pass the memory training: PhyOdtImpedance = 80 60 48 40 PhyDrvImpedancePU = 60 53 48 43 40 36 34 32 30 PhyDrvImpedancePD = 36 34 32 30 28 DramRttNomR0/1 = 240 120 80 60 40 34 DramRttWrR0/1 = 240 120 80 DramRttParkR0/1 = 240 120 80 60 48 40 34 Some of these parameter combinations can be trained through memory and booted to the OS, such as the following group: PhyOdtImpedance = 80 PhyDrvImpedancePU = 60 PhyDrvImpedancePD = 36 DramRttNomR0/1 = 120 DramRttWrR0/1 = 80 DramRttParkR0/1 =40 But after entering the OS, we used the "stream" test tool to test and found that the Memory bandwidth was only 25% of the theoretical value. I suspect that the "user_input_msdg" value I selected affected the stream performance.

  2. where are you measuring your results and how? What we are testing should be the stream performance of the entire link. The following stream testing tools were used: https://www.cs.virginia.edu/stream/ref.html So is there a way to test the bandwidth performance of OCMB>DIMM separately?

  3. When running stream testing under OS, sometimes the following error will be reported and the test will stop. [ 769.211099] Disabling lock debugging due to kernel taint [ 769.211102] MCE: CPU4: machine check (Severe) Host UE Load/Store DAR: 00000000ad56ba80 paddr: 0000000383a30000 [Not recovered] [ 769.211103] MCE: CPU4: PID: 3970 Comm: stream NIP: [00000000100014c0] [ 769.211103] MCE: CPU4: Initiator CPU [ 769.211103] MCE: CPU4: Hardware error [ 769.211104] stream[3970]: unhandled signal 7 at 00000000100014c0 nip 00000000100014c0 lr 0000000010001448 code 4 [ 769.212179] Memory failure: 0x383a3: recovery action for dirty LRU page: Recovered May I ask: Is this also related to the parameter values I selected.

  4. Finally, we currently choose the ' user_input_msdg ' value based on the condition of being able to train in memory and achieve optimal stream performance. Is there a better way to determine which "user_input_msdg" parameter value is optimal for our platform?

dcrowell77 commented 1 year ago

Memory bandwidth was only 25% of the theoretical value.

How do the numbers compare to an equivalent Rainier configuration? Comparing to a real-world example is perhaps more accurate than theory.

  1. That would imply some settings are wrong and the memory is not running correctly. @ecorderoibm can probably answer more intelligently. Do you see any PELs coming from opal-prd in this case?
ecorderoibm commented 1 year ago

those settings you changed don't have any affect on the bandwidth are only for signal integrity on the DRAM interface. I agree with Dan. Would be good to know what you measured using a DDIMM and work backwards from there for comparison and validation of you measurements.

Grubby0624 commented 1 year ago
  1. The following are my test results. Due to the limitations of our platform, I inserted a 3200MHz ISDIMM under the OMI 0A/1A/2A/3A of Proc0. When testing the bandwidth of the DIMM under Proc0, the result was 24558.3MB/s. On Rainier, I inserted a 3200MHz DDIMM under the 0A/1A/2A/3A of Proc0, and the result was 74540.4MB/s. PS: The theoretical bandwidth of four 3200MHz DIMMs is 100GB/s, which has been tested on Rainier and reaches 65% of the theoretical value
  2. When I run the stream script in OS and report an MCE error, I cannot see any PEL in BMC. However, if I copy the script to ‘/etc/profile. d/’ and execute the script automatically during OS startup, the system will report checkststop. I will separate the collected PEL logs into my later answer. Thanks!!
Grubby0624 commented 1 year ago

This is the PEL log: { "Private Header": { "Section Version": "1", "Sub-section type": "0", "Created by": "0xE500", "Created at": "12/21/2022 18:04:46", "Committed at": "12/21/2022 18:04:46", "Creator Subsystem": "BMC", "CSSVER": "", "Platform Log Id": "0x500006C6", "Entry Id": "0x500006C6", "BMC Event Log Id": "2449" }, "User Header": { "Section Version": "1", "Sub-section type": "0", "Log Committed by": "0x2000", "Subsystem": "Memory DIMM", "Event Scope": "Entire Platform", "Event Severity": "Unrecoverable Error", "Event Type": "Not Applicable", "Action Flags": [ "Service Action Required", "Report Externally", "HMC Call Home" ], "Host Transmission": "Not Sent", "HMC Transmission": "Not Sent" }, "Primary SRC": { "Section Version": "1", "Sub-section type": "1", "Created by": "0xE500", "SRC Version": "0x02", "SRC Format": "0x55", "Virtual Progress SRC": "False", "I5/OS Service Event Bit": "False", "Hypervisor Dump Initiated":"False", "Backplane CCIN": "2E2D", "Terminate FW Error": "False", "Deconfigured": "False", "Guarded": "True", "Error Details": { "Message": "Error Signature: 0x60D20020 0x00020003 0xCCAA000E" }, "Valid Word Count": "0x09", "Reference Code": "BD23E510", "Hex Word 2": "00080055", "Hex Word 3": "2E2D0010", "Hex Word 4": "CC009544", "Hex Word 5": "01000000", "Hex Word 6": "60D20020", "Hex Word 7": "00020003", "Hex Word 8": "CCAA000E", "Hex Word 9": "00000000", "Callout Section": { "Callout Count": "1", "Callouts": [{ "FRU Type": "Normal Hardware FRU", "Priority": "Mandatory, replace all with this type as a unit", "Location Code": "U78DA.ND0.WZS01SL-P0-C27", "Part Number": "", "CCIN": "", "Serial Number": "" }] }, "SRC Details": { "Primary Attention": "system checkstop", "Signature Description": { "Chip Desc": "node 0 ocmb 2 (Explorer 2.0)", "Signature": "RDFFIR(0)[14] Mainline read UE", "Attn Type": "recoverable" } } }, "Extended User Header": { "Section Version": "1", "Sub-section type": "0", "Created by": "0x2000", "Reporting Machine Type": "9105-42A", "Reporting Serial Number": "783C4C1", "FW Released Ver": "", "FW SubSys Version": "0.1.1", "Common Ref Time": "00/00/0000 00:00:00", "Symptom Id Len": "36", "Symptom Id": "BD23E510_60D20020_00020003_CCAA000E" }, "Failing MTMS": { "Section Version": "1", "Sub-section type": "0", "Created by": "0x2000", "Machine Type Model": "9105-42A", "Serial Number": "783C4C1" }, "User Data 0": { "Section Version": "1", "Sub-section type": "1", "Created by": "0x2000", "BMCLoad": "1.69 1.39 0.90", "BMCState": "Ready", "BMCUptime": "0y 0d 0h 14m 49s", "BootState": "OSRunning", "ChassisState": "On", "FW Version ID": "0.1.1", "HostState": "Running", "Process Name": "/usr/bin/openpower-hw-diags", "System IM": "60001000" }, "User Data 1": { "Section Version": "1", "Sub-section type": "1", "Created by": "0x2000", "PEL_SUBSYSTEM": "0x23", "SRC6": "1624375328", "SRC7": "131075", "SRC8": "3433693198", "_PID": "1086" }, "User Data 2": { "Section Version": "1", "Sub-section type": "1", "Created by": "0x2000", "Data": [ { "Deconfigured": false, "EntityPath": [ 35, 1, 0, 2, 0, 3, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 ], "GuardType": "GARD_Unrecoverable", "Guarded": true, "LocationCode": "Ufcs-P0-C27", "Priority": "M" } ] }, "User Data 3": { "Section Version": "1", "Sub-section type": "4", "Created by": "0xE500", "Hostboot Scratch Registers": { "0x0000283c": "0xaa811504", "0x000000004602f489": "0x0000000000000000" } }, "User Data 4": { "Section Version": "1", "Sub-section type": "5", "Created by": "0xE500", "Scratch Register Error Signature": { "Chip ID": "0x004b0006", "Signature ID": "0x5993000a" } }, "User Data 5": { "Section Version": "1", "Sub-section type": "3", "Created by": "0xE500", "Callout List FFDC": [ { "Callout Type": "Hardware Callout", "Guard": true, "Priority": "medium", "Target": "physical:sys-0/node-0/dimm-2" } ] }, "User Data 6": { "Section Version": "1", "Sub-section type": "1", "Created by": "0xE500", "Signature List": [ { "Chip Desc": "node 0 proc 0 (P10 2.0)", "Signature": "PB_EXT_FIR(0)[7] pb_x7_fir_err", "Attn Type": "checkstop" }, { "Chip Desc": "node 0 proc 0 (P10 2.0)", "Signature": "MC_DSTL_FIR(1)[1] Subchannel A AFU initiated Recoverable Attention", "Attn Type": "recoverable" }, { "Chip Desc": "node 0 proc 0 (P10 2.0)", "Signature": "MC_DSTL_FIR(2)[1] Subchannel A AFU initiated Recoverable Attention", "Attn Type": "recoverable" }, { "Chip Desc": "node 0 ocmb 2 (Explorer 2.0)", "Signature": "RDFFIR(0)[0] Mainline read MPE on rank 0", "Attn Type": "recoverable" }, { "Chip Desc": "node 0 ocmb 2 (Explorer 2.0)", "Signature": "RDFFIR(0)[1] Mainline read MPE on rank 1", "Attn Type": "recoverable" }, { "Chip Desc": "node 0 ocmb 2 (Explorer 2.0)", "Signature": "RDFFIR(0)[8] Mainline read NCE", "Attn Type": "recoverable" }, { "Chip Desc": "node 0 ocmb 2 (Explorer 2.0)", "Signature": "RDFFIR(0)[14] Mainline read UE", "Attn Type": "recoverable" }, { "Chip Desc": "node 0 ocmb 2 (Explorer 2.0)", "Signature": "RDFFIR(0)[17] Mainline read IUE", "Attn Type": "recoverable" }, { "Chip Desc": "node 0 ocmb 2 (Explorer 2.0)", "Signature": "RDFFIR(0)[19] Mainline read IMPE", "Attn Type": "recoverable" }, { "Chip Desc": "node 0 ocmb 4 (Explorer 2.0)", "Signature": "RDFFIR(0)[0] Mainline read MPE on rank 0", "Attn Type": "recoverable" }, { "Chip Desc": "node 0 ocmb 4 (Explorer 2.0)", "Signature": "RDFFIR(0)[1] Mainline read MPE on rank 1", "Attn Type": "recoverable" }, { "Chip Desc": "node 0 ocmb 4 (Explorer 2.0)", "Signature": "RDFFIR(0)[8] Mainline read NCE", "Attn Type": "recoverable" }, { "Chip Desc": "node 0 ocmb 4 (Explorer 2.0)", "Signature": "RDFFIR(0)[9] Mainline read TCE", "Attn Type": "recoverable" }, { "Chip Desc": "node 0 ocmb 4 (Explorer 2.0)", "Signature": "RDFFIR(0)[14] Mainline read UE", "Attn Type": "recoverable" }, { "Chip Desc": "node 0 ocmb 4 (Explorer 2.0)", "Signature": "RDFFIR(0)[17] Mainline read IUE", "Attn Type": "recoverable" }, { "Chip Desc": "node 0 ocmb 4 (Explorer 2.0)", "Signature": "RDFFIR(0)[19] Mainline read IMPE", "Attn Type": "recoverable" }, { "Chip Desc": "node 0 proc 1 (P10 2.0)", "Signature": "PB_EXT_FIR(0)[6] pb_x6_fir_err", "Attn Type": "checkstop" }, { "Chip Desc": "node 0 proc 1 (P10 2.0)", "Signature": "MC_DSTL_FIR(5)[1] Subchannel A AFU initiated Recoverable Attention", "Attn Type": "recoverable" }, { "Chip Desc": "node 0 proc 2 (P10 2.0)", "Signature": "PB_EXT_FIR(0)[2] pb_x2_fir_err", "Attn Type": "checkstop" }, { "Chip Desc": "node 0 proc 2 (P10 2.0)", "Signature": "MC_DSTL_FIR(1)[1] Subchannel A AFU initiated Recoverable Attention", "Attn Type": "recoverable" }, { "Chip Desc": "node 0 proc 2 (P10 2.0)", "Signature": "MC_DSTL_FIR(2)[1] Subchannel A AFU initiated Recoverable Attention", "Attn Type": "recoverable" }, { "Chip Desc": "node 0 proc 3 (P10 2.0)", "Signature": "N1_LOCAL_FIR(0)[60] Hypervisor / Hostboot / OPAL initiated Terminate Immediate", "Attn Type": "checkstop" }, { "Chip Desc": "node 0 proc 3 (P10 2.0)", "Signature": "MC_DSTL_FIR(5)[1] Subchannel A AFU initiated Recoverable Attention", "Attn Type": "recoverable" }, { "Chip Desc": "node 0 proc 3 (P10 2.0)", "Signature": "MC_DSTL_FIR(6)[1] Subchannel A AFU initiated Recoverable Attention", "Attn Type": "recoverable" }, { "Chip Desc": "node 0 proc 3 (P10 2.0)", "Signature": "MC_DSTL_FIR(6)[2] Subchannel A AFU initiated Special Attention", "Attn Type": "host attention" }, { "Chip Desc": "node 0 ocmb 58 (Explorer 2.0)", "Signature": "RDFFIR(0)[0] Mainline read MPE on rank 0", "Attn Type": "recoverable" }, { "Chip Desc": "node 0 ocmb 58 (Explorer 2.0)", "Signature": "RDFFIR(0)[1] Mainline read MPE on rank 1", "Attn Type": "recoverable" }, { "Chip Desc": "node 0 ocmb 58 (Explorer 2.0)", "Signature": "RDFFIR(0)[8] Mainline read NCE", "Attn Type": "recoverable" }, { "Chip Desc": "node 0 ocmb 58 (Explorer 2.0)", "Signature": "RDFFIR(0)[9] Mainline read TCE", "Attn Type": "recoverable" }, { "Chip Desc": "node 0 ocmb 58 (Explorer 2.0)", "Signature": "RDFFIR(0)[14] Mainline read UE", "Attn Type": "recoverable" }, { "Chip Desc": "node 0 ocmb 58 (Explorer 2.0)", "Signature": "RDFFIR(0)[17] Mainline read IUE", "Attn Type": "recoverable" }, { "Chip Desc": "node 0 ocmb 58 (Explorer 2.0)", "Signature": "RDFFIR(0)[19] Mainline read IMPE", "Attn Type": "recoverable" }, { "Chip Desc": "node 0 ocmb 58 (Explorer 2.0)", "Signature": "RDFFIR(0)[37] Maintenance IUE", "Attn Type": "recoverable" } ] }, "User Data 7": { "Section Version": "1", "Sub-section type": "2", "Created by": "0xE500", "Register Dump": [ "node 0 proc 0 (P10 2.0) ****", " GFIR_CS (0x570F001C) 1000 0000 0000 0000", " CFIR_N1_CS (0x03040000) 8000 0000 4000 0000", " CFIR_N1_CS_MASK (0x03040040) 2000 0000 0000 0000", " PB_EXT_FIR (0x030113AE) 0100 0000 0000 0000", " PB_EXT_FIR_MASK (0x030113B1) D400 0000 0000 0000", " GFIR_RE (0x570F001B) 000C 0000 0000 0000", " CFIR_MC_RE (0x0C040001) 8100 0000 0000 0000", " MC_DSTL_CFG2 (0x0C010D4E) 0000 0000 0600 0000", " MC_DSTL_FIR (0x0C010D40) 4000 0000 0100 0000", " MC_DSTL_FIR_MASK (0x0C010D43) 1F01 3C00 6780 0000", " MC_DSTL_FIR_ACT0 (0x0C010D46) 880C C300 1800 0000", " MC_DSTL_FIR_ACT1 (0x0C010D47) CC2F C363 9800 0000", " MC_DSTL_FIR_ACT2 (0x0C010D49) 2200 0000 0000 0000", " CFIR_MC_RE (0x0D040001) 8400 0000 0000 0000", " MC_DSTL_CFG2 (0x0D010D0E) 0000 0000 0600 0000", " MC_DSTL_FIR (0x0D010D00) 4000 0000 0100 0000", " MC_DSTL_FIR_MASK (0x0D010D03) 1F01 3C00 6780 0000", " MC_DSTL_FIR_ACT0 (0x0D010D06) 880C C300 1800 0000", " MC_DSTL_FIR_ACT1 (0x0D010D07) CC2F C363 9800 0000", " MC_DSTL_FIR_ACT2 (0x0D010D09) 2200 0000 0000 0000", "node 0 ocmb 6 (Explorer 2.0) ", " CHIPLET_OCMB_FIR_MASK (0x08040002) 6627 FFE0 0000 0000", "node 0 ocmb 0 (Explorer 2.0) ", " CHIPLET_OCMB_FIR_MASK (0x08040002) 6627 FFE0 0000 0000", "node 0 ocmb 4 (Explorer 2.0) ", " CHIPLET_OCMB_FIR_MASK (0x08040002) 6627 FFE0 0000 0000", " CHIPLET_OCMB_RE_FIR (0x08040001) 0100 0000 0000 0000", " FARB0 (0x08011415) 0000 9880 8200 5D0C", " MBSEC0 (0x08011855) 0000 0000 0FFF FFF0", " MBSEC1 (0x08011856) FFF0 0000 0000 0000", " MBSTR (0x08011857) 000F FFFF 3400 01C0", " MBNCER (0x0801186A) 007B 2C36 AC00 0000", " MBRCER (0x0801186B) 008E 0C2D AC00 0000", " MBMPER (0x0801186C) 08B2 FFB9 C800 0000", " MBUER (0x0801186D) 008E 0C2D AC00 0000", " MC_ADDR_TRANS0 (0x0801186F) 8206 4000 3DE6 3DCD", " MC_ADDR_TRANS1 (0x08011870) 100F 0000 2708 090A", " MC_ADDR_TRANS2 (0x08011871) 0B0C 0504 0003 0200", " MBSEVR0 (0x0801187E) CA01 0000 0000 0000", " MCBAGRA (0x080118D6) 0020 0000 0000 0000", " MCBMCAT (0x080118D7) 0004 DE8E 4400 0000", " MCB_CNTLSTAT (0x080118DC) 8000 0000 0000 0000", " MCBCFG (0x080118E0) 0000 0000 0000 0020", " EXP_MSR (0x08011C0C) 0051 2000 0000 0000", " RDF_ERR_RPT_1 (0x08011C0F) 0000 3000 0000 0000", " HW_MS (0x08011C10) 51C0 0000 0000 0000", " HW_MS (0x08011C11) 51C0 0000 0000 0000", " RDFFIR (0x08011C00) C0D2 5001 0000 0000", " RDFFIR_MASK (0x08011C03) 0038 0FFF A000 7FFF", " RDFFIR_ACT1 (0x08011C07) FFC3 7000 17E6 0000", "node 0 ocmb 2 (Explorer 2.0) ", " CHIPLET_OCMB_FIR_MASK (0x08040002) 6627 FFE0 0000 0000", " CHIPLET_OCMB_RE_FIR (0x08040001) 0100 0000 0000 0000", " FARB0 (0x08011415) 0000 9880 8200 5D0C", " MBSEC0 (0x08011855) 0000 0000 0101 0510", " MBSEC1 (0x08011856) A4E0 0000 0000 0000", " MBSTR (0x08011857) 000F FFFF 3400 01C0", " MBNCER (0x0801186A) 086A F836 A400 0000", " MBRCER (0x0801186B) 086A F836 A400 0000", " MBMPER (0x0801186C) 004B 922C 0800 0000", " MBUER (0x0801186D) 0897 7822 C400 0000", " MC_ADDR_TRANS0 (0x0801186F) 8206 4000 3DE6 3DCD", " MC_ADDR_TRANS1 (0x08011870) 100F 0000 2708 090A", " MC_ADDR_TRANS2 (0x08011871) 0B0C 0504 0003 0200", " MBSEVR0 (0x0801187E) 8B01 0000 0000 0000", " MCBAGRA (0x080118D6) 0020 0000 0000 0000", " MCBMCAT (0x080118D7) 0004 DE9A 2000 0000", " MCB_CNTLSTAT (0x080118DC) 8000 0000 0000 0000", " MCBCFG (0x080118E0) 0000 0000 0000 0020", " EXP_MSR (0x08011C0C) 0098 0000 0000 0000", " RDF_ERR_RPT_1 (0x08011C0F) 0000 3000 0000 0000", " HW_MS (0x08011C10) 98C0 0000 0000 0000", " HW_MS (0x08011C11) 98C0 0000 0000 0000", " RDFFIR (0x08011C00) C092 5001 0000 0000", " RDFFIR_MASK (0x08011C03) 0038 0FFF A000 7FFF", " RDFFIR_ACT1 (0x08011C07) FFC3 7000 17E6 0000", "node 0 proc 1 (P10 2.0) ****", " GFIR_CS (0x570F001C) 1000 0000 0000 0000", " CFIR_N1_CS (0x03040000) 8000 0000 4000 0000", " CFIR_N1_CS_MASK (0x03040040) 2000 0000 0000 0000", " PB_EXT_FIR (0x030113AE) 0200 0000 0000 0000", " PB_EXT_FIR_MASK (0x030113B1) B400 0000 0000 0000", " GFIR_RE (0x570F001B) 0002 0000 0000 0000", " CFIR_MC_RE (0x0E040001) 8100 0000 0000 0000", " MC_DSTL_CFG2 (0x0E010D4E) 0000 0000 0600 0000", " MC_DSTL_FIR (0x0E010D40) 4000 0000 0100 0000", " MC_DSTL_FIR_MASK (0x0E010D43) 1F01 3C00 6780 0000", " MC_DSTL_FIR_ACT0 (0x0E010D46) 880C C300 1800 0000", " MC_DSTL_FIR_ACT1 (0x0E010D47) CC2F C363 9800 0000", " MC_DSTL_FIR_ACT2 (0x0E010D49) 2200 0000 0000 0000", "node 0 proc 2 (P10 2.0) ****", " GFIR_CS (0x570F001C) 1000 0000 0000 0000", " CFIR_N1_CS (0x03040000) 8000 0000 4000 0000", " CFIR_N1_CS_MASK (0x03040040) 2000 0000 0000 0000", " PB_EXT_FIR (0x030113AE) 2000 0000 0000 0000", " PB_EXT_FIR_MASK (0x030113B1) D400 0000 0000 0000", " GFIR_RE (0x570F001B) 000C 0000 0000 0000", " CFIR_MC_RE (0x0C040001) 8100 0000 0000 0000", " MC_DSTL_CFG2 (0x0C010D4E) 0000 0000 0600 0000", " MC_DSTL_FIR (0x0C010D40) 4000 0000 0100 0000", " MC_DSTL_FIR_MASK (0x0C010D43) 1F01 3C00 6780 0000", " MC_DSTL_FIR_ACT0 (0x0C010D46) 880C C300 1800 0000", " MC_DSTL_FIR_ACT1 (0x0C010D47) CC2F C363 9800 0000", " MC_DSTL_FIR_ACT2 (0x0C010D49) 2200 0000 0000 0000", " CFIR_MC_RE (0x0D040001) 8400 0000 0000 0000", " MC_DSTL_CFG2 (0x0D010D0E) 0000 0000 0600 0000", " MC_DSTL_FIR (0x0D010D00) 4000 0000 0100 0000", " MC_DSTL_FIR_MASK (0x0D010D03) 1F01 3C00 6780 0000", " MC_DSTL_FIR_ACT0 (0x0D010D06) 880C C300 1800 0000", " MC_DSTL_FIR_ACT1 (0x0D010D07) CC2F C363 9800 0000", " MC_DSTL_FIR_ACT2 (0x0D010D09) 2200 0000 0000 0000", "node 0 proc 3 (P10 2.0) ****", " GFIR_CS (0x570F001C) 1000 0000 0000 0000", " CFIR_N1_CS (0x03040000) 8800 0000 0000 0000", " CFIR_N1_CS_MASK (0x03040040) 2000 0000 0000 0000", " N1_LOCAL_FIR (0x03040100) 0000 0000 0000 0008", " N1_LOCAL_FIR_MASK (0x03040103) 00EF F3FF FFFF FF07", " N1_LOCAL_FIR_ACT1 (0x03040107) FFFF FFFF FFFF FF07", " GFIR_RE (0x570F001B) 0003 0000 0000 0000", " CFIR_MC_RE (0x0E040001) 8100 0000 0000 0000", " MC_DSTL_CFG2 (0x0E010D4E) 0000 0000 0600 0000", " MC_DSTL_FIR (0x0E010D40) 4000 0000 0100 0000", " MC_DSTL_FIR_MASK (0x0E010D43) 1F01 3C00 6780 0000", " MC_DSTL_FIR_ACT0 (0x0E010D46) 880C C300 1800 0000", " MC_DSTL_FIR_ACT1 (0x0E010D47) CC2F C363 9800 0000", " MC_DSTL_FIR_ACT2 (0x0E010D49) 2200 0000 0000 0000", " CFIR_MC_RE (0x0F040001) 8400 0000 0000 0000", " MC_DSTL_CFG2 (0x0F010D0E) 0000 0000 0600 0000", " MC_DSTL_FIR (0x0F010D00) 6000 0000 0100 0000", " MC_DSTL_FIR_MASK (0x0F010D03) 1F01 3C00 6780 0000", " MC_DSTL_FIR_ACT0 (0x0F010D06) 880C C300 1800 0000", " MC_DSTL_FIR_ACT1 (0x0F010D07) CC2F C363 9800 0000", " MC_DSTL_FIR_ACT2 (0x0F010D09) 2200 0000 0000 0000", " GFIR_HA (0x570F002B) 0001 0000 0000 0000", " CFIR_MC_HA (0x0F040004) 8400 0000 0000 0000", "node 0 ocmb 58 (Explorer 2.0) **", " CHIPLET_OCMB_FIR_MASK (0x08040002) 6627 FFE0 0000 0000", " CHIPLET_OCMB_RE_FIR (0x08040001) 0100 0000 0000 0000", " FARB0 (0x08011415) 0000 9880 8200 5D0C", " MBSEC0 (0x08011855) 0000 0000 020E 0B50", " MBSEC1 (0x08011856) FFF0 0000 0000 0020", " MBSTR (0x08011857) 000F FFFF 3400 01C0", " MBNCER (0x0801186A) 008E 163E C000 0000", " MBRCER (0x0801186B) 0032 9E22 4800 0000", " MBMPER (0x0801186C) 08AA 1030 2000 0000", " MBUER (0x0801186D) 0010 A03C 2800 0000", " MC_ADDR_TRANS0 (0x0801186F) 8206 4000 3DE6 3DCD", " MC_ADDR_TRANS1 (0x08011870) 100F 0000 2708 090A", " MC_ADDR_TRANS2 (0x08011871) 0B0C 0504 0003 0200", " MBSEVR0 (0x0801187E) 5101 0000 0000 0000", " MCBAGRA (0x080118D6) 0020 0000 0000 0000", " MCBMCAT (0x080118D7) 0004 E201 CC00 0000", " MCB_CNTLSTAT (0x080118DC) 8000 0000 0000 0000", " MCBCFG (0x080118E0) 0000 0000 0000 0020", " EXP_MSR (0x08011C0C) 00CA 2000 0000 0000", " RDF_ERR_RPT_1 (0x08011C0F) 0000 3000 0000 0000", " HW_MS (0x08011C10) CAC0 0000 0000 0000", " HW_MS (0x08011C11) CAC0 0000 0000 0000", " RDFFIR (0x08011C00) C0DA 5009 0400 0000", " RDFFIR_MASK (0x08011C03) 0038 0FFF A000 7FFF", " RDFFIR_ACT1 (0x08011C07) FFC3 7000 17E6 0000", "node 0 ocmb 62 (Explorer 2.0) **", " CHIPLET_OCMB_FIR_MASK (0x08040002) 6627 FFE0 0000 0000" ] }, "User Data 8": { "Section Version": "1", "Sub-section type": "1", "Created by": "0x2000", "PEL Internal Debug Data": { "SRC": [ "No VPD found for /xyz/openbmc_project/inventory/system/chassis/motherboard/dimm2: sd_bus_call: xyz.openbmc_project.Common.Error.ResourceNotFound: The resource is not found." ] } } }

ecorderoibm commented 1 year ago

It appears you are taking a lot of memory errors. I would re-evaluate the settings you are using for the DDR interface. I assume you have done signal integrity simulations and have determined those settings. Another option would be to probe them and see what quality the bus appears to have. I can't tell you what to put there, because it is very particular to your design and implementation

Grubby0624 commented 1 year ago
  1. Today, I limited the memory frequency to 2666MHz (of course, I also used the SI parameter of 2666). When entering the OS, I was able to run the Stream test, andthere were no MCE errors, but the results were also unqualified: For example, there are four 2666MHz cables under Proc0, with a bandwidth of only about 20000MB/s I currently want to address the issue of stream testing reporting MCE at 3200MHz. Based on the absence of MCE errors at 2666MHz, I think this may be an SI issue. I have many 3200MHz parameter combinations that can be used for training, and I plan to try running streams one by one to see if there are MCE errors. May I ask if there is a better method, or do you suspect that this may be related to which parameter settings? This way, I can focus on and test this parameter.
  2. I received an SI simulation report from my colleague, and I understand that the recommended value for this ODT is related to DDR: Operation type Memory Controller configuration DRAM configuration Write pm27_dq_d2828 DQ_ODT_80 Read pm27_dq_t48 DQ_RON_34_ODT_OFF Address pm27_ca_d40_srf_c3p DCA_IBT_100 But I'm not sure if what we mean by 'signal integrity simulations' is the same thing. sorry, Because I haven't read such simulation reports before, I'm not sure which results to focus on~
Grubby0624 commented 1 year ago

I change the IO OMI CHANNEL default value to LOSS to LOW_ LOSS, and then using different SI parameters on two different sets of DIMMs, the memory can now boot and run Streams without reporting MCE errors. Unfortunately, the Stream performance still does not meet the requirements

Grubby0624 commented 1 year ago

  1. I only configure one DIMM under each Proc, and then specify 1/2/12 cores to run the Stream. The resulting values are set to 6000MB/s (the theoretical value should be 25GB/s)

  2. When specifying a Core to run a Stream, I use the top command to check the CPU usage. The Stream application occupies 100% of the CPU

  3. I put the ARRAY of the stream_ When the SIZE is downgraded and paired with 128K, the measured bandwidth of a core is approximately 490GB/s. I think this should be a memory operation directly completed in the cache.

May I ask:

  1. At this point, there are no PEL or other error reports. Can I assume that the memory subsystem has not reported any errors? If so, how should we monitor it?
  2. Regarding my above experimental phenomenon, can I lock the issue onto the OMI -->OCMB -->DDR link? What I mean is to exclude the impact of certain errors within the Processor?
dcrowell77 commented 1 year ago

1 - Yes, if there are no PELs then there have been no errors detected. 2 - I don't see any indication of "errors" so this is all just hardware tuning.

Have you been in contact with your IBM contacts regarding how the theoretical versus actual numbers relate from a hardware perspective? The Hostboot team is not directly involved in any of the hardware tuning or performance measurements.

@ecorderoibm @steffenchris - Anything to add?

Grubby0624 commented 1 year ago

Regarding the issue of Stream performance not meeting standards, we found that there was an issue with the I2C address configuration of our platform's Thermal Sensor. We sent 'EXPFW_TEMP_SENSOR_CONFIG_INTERVALREAD' to OCMB, but OCMB was unable to read a valid sensor value through the corresponding I2C address: If we don't send EXP FW TEMP SENSOR CONFIG INTERVAL_ READ to OCMB, and the results of the Stream test meet the standards. https://github.com/open-power/ocmb-explorer-fw/blob/33e3301f619163c8d3aad373b045cb8db494d3d1/_exp/src/temp_sensor/temp_sensor_plat.c#L564C19-L564C19

By the way: I guess OCMB will enter a "safe mode" if it cannot read the Thermal Sensor, but there is no relevant introduction found in the code and Spec. Can you provide a professional explanation? Or do you know where I can get relevant explanations?

Grubby0624 commented 1 year ago

Unfortunately, we encountered same MCE error while running HTX:

  1. If we specify the frequency to 3200MHz, running mdt.mem for less than 10 minutes will result in MCE error. If we specify the frequency to 2666MHz, there will be no MCE error even after running for 40 minutes. So I guess it's still a problem of Signal integrity
  2. Specify 3200MHz for Stream testing on the same platform and no MCE issues will be reported again After an MCE error occurred, we triggered checkstop to collect FIRdata by injecting error to TI, and found that there were 2 MCs in each Proc that always reported the following error: { "Chip Desc": "node 0 proc 0 (P10 2.0)", "Signature": "MC_DSTL_FIR(1)[1] Subchannel A AFU initiated Recoverable Attention", "Attn Type": "recoverable" }, And when trying on different platforms, the MC numbers that reported errors are consistent. Regarding this error, I would like to ask:
  3. Based on the error content, can you determine whether this is an SI issue with the DDR bus or the OMI bus?
  4. What is the "AFU" mentioned here? Regarding this MCE error, I plan to adjust the SI parameters of DDR and OMI. I would like to confirm what the following two OMI attributes mean: IO OMI PRE1 IO OMI PRE2 Our current values for both values are 0. Do you have any other debugging suggestions?
dcrowell77 commented 1 year ago

I guess OCMB will enter a "safe mode" if it cannot read the Thermal Sensor

This is exactly true. There are actually too layers of throttles possible. First, the OCC manages overall system performance based on the temperature data it gets back from Explorer. Second, there is a deadman timer in Explorer itself that will go to the emergency throttles if it isn't periodically reset by the OCC. I don't have a pointer to a spec but searching for FARB4Q and FARB3Q in the hostboot and occ code will find some of the relevant code. I suspect that your i2c errors led to OCC going into safe mode which would have a severe effect on system performance. There should be error logs for that, assuming that you have hbrt/opal-prd completely working under opal.

dcrowell77 commented 1 year ago

IO OMI PRE1 IO OMI PRE2

I can't personally tell you exactly what these are but I can say that we have them both set to zero on all of our boxes.

dcrowell77 commented 1 year ago

AFU = Accelerator Function Unit. It was left over from OpenCAPI. It is still referenced throughout OMI since its all OpenCAPI based.

Grubby0624 commented 12 months ago

Thank you for your answer. I now have some concepts about the problems I am facing. So this is definitely used on the OMI interface, we cannot disable this module? My concern is that there are issues with both our OMI and DDR bus SI. I think I know how to adjust the SI parameters of DDR, but what are the SI parameters of OMI? Can we also adjust it?

steffenchris commented 12 months ago

These values control the OMI Tx FFE. Where each value is X/128 precursor. For example, if IO_OMIPRE1=8. Then 8/128=6.25% of Tx FFE Precursor1. IO OMI PRE1 IO OMI_ PRE2

dcrowell77 commented 12 months ago

we cannot disable this module?

Not sure what you mean by "module" here. The OMI bus is obviously fundamental to the system design. You could try masking off the FIRs that are turning on but I suspect that will just hide the issue and you'll fail in some other way.

Grubby0624 commented 11 months ago

I put PHY The INIT mode has been set to WRITE EYE Train, I was puzzled by the printed results. Taking Rank0's bit0 as an example, from these logs, I understand that Vref values around 50 should be selected to ensure that as many Delays as possible can pass: 163.65588|ocmb:k0:n0:s0:p26 VrefDQ RANK 0, DBYTE 0 BIT 0 EYE INDEX 0 -- MIN: 127 MAX: 0 163.65589|ocmb:k0:n0:s0:p26 VrefDQ RANK 0, DBYTE 0 BIT 0 EYE INDEX 1 -- MIN: 127 MAX: 0 163.65590|ocmb:k0:n0:s0:p26 VrefDQ RANK 0, DBYTE 0 BIT 0 EYE INDEX 2 -- MIN: 44 MAX: 50 163.65591|ocmb:k0:n0:s0:p26 VrefDQ RANK 0, DBYTE 0 BIT 0 EYE INDEX 3 -- MIN: 40 MAX: 55 163.65592|ocmb:k0:n0:s0:p26 VrefDQ RANK 0, DBYTE 0 BIT 0 EYE INDEX 4 -- MIN: 34 MAX: 59 163.65593|ocmb:k0:n0:s0:p26 VrefDQ RANK 0, DBYTE 0 BIT 0 EYE INDEX 5 -- MIN: 33 MAX: 62 163.65595|ocmb:k0:n0:s0:p26 VrefDQ RANK 0, DBYTE 0 BIT 0 EYE INDEX 6 -- MIN: 29 MAX: 66 163.65596|ocmb:k0:n0:s0:p26 VrefDQ RANK 0, DBYTE 0 BIT 0 EYE INDEX 7 -- MIN: 23 MAX: 73 163.65597|ocmb:k0:n0:s0:p26 VrefDQ RANK 0, DBYTE 0 BIT 0 EYE INDEX 8 -- MIN: 21 MAX: 73 163.65598|ocmb:k0:n0:s0:p26 VrefDQ RANK 0, DBYTE 0 BIT 0 EYE INDEX 9 -- MIN: 17 MAX: 73 163.65599|ocmb:k0:n0:s0:p26 VrefDQ RANK 0, DBYTE 0 BIT 0 EYE INDEX 10 -- MIN: 18 MAX: 73 163.65600|ocmb:k0:n0:s0:p26 VrefDQ RANK 0, DBYTE 0 BIT 0 EYE INDEX 11 -- MIN: 18 MAX: 73 163.65602|ocmb:k0:n0:s0:p26 VrefDQ RANK 0, DBYTE 0 BIT 0 EYE INDEX 12 -- MIN: 17 MAX: 73 163.65603|ocmb:k0:n0:s0:p26 VrefDQ RANK 0, DBYTE 0 BIT 0 EYE INDEX 13 -- MIN: 18 MAX: 73 163.65604|ocmb:k0:n0:s0:p26 VrefDQ RANK 0, DBYTE 0 BIT 0 EYE INDEX 14 -- MIN: 17 MAX: 73 163.65605|ocmb:k0:n0:s0:p26 VrefDQ RANK 0, DBYTE 0 BIT 0 EYE INDEX 15 -- MIN: 17 MAX: 73 163.65606|ocmb:k0:n0:s0:p26 VrefDQ RANK 0, DBYTE 0 BIT 0 EYE INDEX 16 -- MIN: 17 MAX: 73 163.65607|ocmb:k0:n0:s0:p26 VrefDQ RANK 0, DBYTE 0 BIT 0 EYE INDEX 17 -- MIN: 18 MAX: 73 163.65609|ocmb:k0:n0:s0:p26 VrefDQ RANK 0, DBYTE 0 BIT 0 EYE INDEX 18 -- MIN: 20 MAX: 73 163.65610|ocmb:k0:n0:s0:p26 VrefDQ RANK 0, DBYTE 0 BIT 0 EYE INDEX 19 -- MIN: 22 MAX: 71 163.65611|ocmb:k0:n0:s0:p26 VrefDQ RANK 0, DBYTE 0 BIT 0 EYE INDEX 20 -- MIN: 23 MAX: 70 163.65612|ocmb:k0:n0:s0:p26 VrefDQ RANK 0, DBYTE 0 BIT 0 EYE INDEX 21 -- MIN: 24 MAX: 68 163.65613|ocmb:k0:n0:s0:p26 VrefDQ RANK 0, DBYTE 0 BIT 0 EYE INDEX 22 -- MIN: 27 MAX: 66 163.65614|ocmb:k0:n0:s0:p26 VrefDQ RANK 0, DBYTE 0 BIT 0 EYE INDEX 23 -- MIN: 32 MAX: 63 163.65616|ocmb:k0:n0:s0:p26 VrefDQ RANK 0, DBYTE 0 BIT 0 EYE INDEX 24 -- MIN: 33 MAX: 60 163.65617|ocmb:k0:n0:s0:p26 VrefDQ RANK 0, DBYTE 0 BIT 0 EYE INDEX 25 -- MIN: 38 MAX: 56 163.65618|ocmb:k0:n0:s0:p26 VrefDQ RANK 0, DBYTE 0 BIT 0 EYE INDEX 26 -- MIN: 39 MAX: 55 163.65619|ocmb:k0:n0:s0:p26 VrefDQ RANK 0, DBYTE 0 BIT 0 EYE INDEX 27 -- MIN: 45 MAX: 52 163.65620|ocmb:k0:n0:s0:p26 VrefDQ RANK 0, DBYTE 0 BIT 0 EYE INDEX 28 -- MIN: 127 MAX: 0 163.65621|ocmb:k0:n0:s0:p26 VrefDQ RANK 0, DBYTE 0 BIT 0 EYE INDEX 29 -- MIN: 127 MAX: 0 163.65623|ocmb:k0:n0:s0:p26 VrefDQ RANK 0, DBYTE 0 BIT 0 EYE INDEX 30 -- MIN: 127 MAX: 0 But it ultimately selects VrefDQ Center is 22: 163.78035|ocmb:k0:n0:s0:p26 VrefDQ_Center RANK 0, NIBBLE 0: 22

Grubby0624 commented 11 months ago

May I ask:

  1. Is there a deviation in my understanding of the training log? Why is the training result different from what I expected?
  2. InitVrefDQ [0] is a parameter that affects WRITE EYE Does the process of training have any impact? I have modified this value, does the training result seem to have not changed?
  3. If my previous understanding of the training results is correct, how should I change the Vref training results to around 50?
  4. If PHY The mode of INIT is DEFAULT Train, how is its Vref determined?
Grubby0624 commented 11 months ago

May I ask:

  1. Is there a deviation in my understanding of the training log? Why is the training result different from what I expected?
  2. InitVrefDQ [0] is a parameter that affects WRITE EYE Does the process of training have any impact? I have modified this value, does the training result seem to have not changed?
  3. If my previous understanding of the training results is correct, how should I change the Vref training results to around 50?
  4. If PHY The mode of INIT is DEFAULT Train, how is its Vref determined?
dcrowell77 commented 11 months ago

@ecorderoibm @esteban012 can maybe weigh in on these questions. You may need to engage Microchip as well since their firmware directly drives the DDR training, not Hostboot.

ecorderoibm commented 11 months ago

The initVrefDQ setting will only tell the algorithm where to start. it really should have no impact on the final results. I think your opinion on the results is correct. The question as to why it selects a lower number than you expect, I would ask to MCHP. I am not sure where to look to know why, they will probably know. They will also be able to tell you how to override the training results if you are looking to try it at a different set point. Same for question 4 I think you can ask them to take you through the training algorithm, which will also be part of understanding why the Vref result was selected.

sglancy6 commented 11 months ago
  1. I am not sure of the details used. Microchip would need to respond to specific questions regarding the draminit training algorithm. However, you did not post the TxDqDly_Center. Could you please post the results for TxDqDly_Center for the bit mentioned?
  2. InitVrefDQ is the initial VREF that is used for the write eye training. The algorithm should then train the WR VREF and write delays after using this setting as a starting point. If you want to skip WR VREF training,
  3. To update the WR VREF results, you would need to use CCS to issue the WR VREF latch commands on a per DRAM basis. Please consult the generic/memory/lib/dimm/ddr4/pda.H for the API on how to handle the latching commands on a per DRAM basis
  4. I am not sure of the details used. Microchip would need to respond to specific questions regarding the draminit training algorithm.
Grubby0624 commented 11 months ago
  1. The TxDqDly_Center is 15
  2. When conducting MCBIST testing, we found that the memory that reported MCE errors would have the following errors: 2023-08-11T15:02:29.532101698,[INF]:#### MCBIST: explorer:k0:n0:s0:p04 #### 2023-08-11T15:02:29.532117806,[INF]:########## PORT: 0 ############# 2023-08-11T15:02:29.532120243,[INF]:Byte: 0011223344556677 8899 2023-08-11T15:02:29.532121884,[INF]:Nibble: 0101010101010101 0101 2023-08-11T15:02:29.532123572,[INF]:### DIMM: 0 ### 2023-08-11T15:02:29.532125242,[INF]:Mask :0000000000000000 00ff 2023-08-11T15:02:29.532126816,[INF]:R0 Err Count:f............... ..MM 2023-08-11T15:02:29.532128386,[INF]:R0 Err Flags:X............... ..MM 2023-08-11T15:02:29.532129943,[INF]:R1 Err Count:ff.............. ..MM 2023-08-11T15:02:29.532131483,[INF]:R1 Err Flags:XX.............. ..MM I will paste the corresponding error trap on the next reply. Regarding this error report, I would like to consult:
    1. Is this error related to MCE errors? Because we found that the memory location where the MCE error was reported corresponds to the memory location where this error was reported in MCBIST.
    2. What does MCBIST specifically do? I understand that it is to write a fixed pattern into DRAM through a method called super fast, and then read it out to determine whether the writing was successful? But what addresses have been written, and how is it determined whether they are successful?
    3. Are there any other suggestions from this log? I'm not sure if the MCE error is a signal integrity issue now?
Grubby0624 commented 11 months ago

This is the error trap:2023-08-11T15:02:53.681864530,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 0 2023-08-11T15:02:53.681882828,[INF]: subtest: 3 2023-08-11T15:02:53.681885395,[INF]: subcommand: 0 2023-08-11T15:02:53.681887183,[INF]: dimm: 0 2023-08-11T15:02:53.681888927,[INF]: mrank: 1 2023-08-11T15:02:53.681890648,[INF]: srank: 0 2023-08-11T15:02:53.681892342,[INF]: BG: 2 2023-08-11T15:02:53.681894021,[INF]: BA: 1 2023-08-11T15:02:53.681895952,[INF]: row: 0x1d914 2023-08-11T15:02:53.681897738,[INF]: col: 0x05 2023-08-11T15:02:53.681899540,[INF]: beat pair: 6/7 2023-08-11T15:02:53.681901165,[INF]: error mode: 2 2023-08-11T15:02:53.681904220,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.681905897,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.681907560,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.681909494,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.681911381,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.681913224,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.681914991,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.681918149,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 1 2023-08-11T15:02:53.681919897,[INF]: subtest: 3 2023-08-11T15:02:53.681921533,[INF]: subcommand: 0 2023-08-11T15:02:53.681923164,[INF]: dimm: 0 2023-08-11T15:02:53.681924747,[INF]: mrank: 1 2023-08-11T15:02:53.681926360,[INF]: srank: 0 2023-08-11T15:02:53.681927967,[INF]: BG: 3 2023-08-11T15:02:53.681929564,[INF]: BA: 0 2023-08-11T15:02:53.681931216,[INF]: row: 0x135cd 2023-08-11T15:02:53.681932856,[INF]: col: 0x25 2023-08-11T15:02:53.681934476,[INF]: beat pair: 6/7 2023-08-11T15:02:53.681936067,[INF]: error mode: 2 2023-08-11T15:02:53.681938142,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.681939757,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.681941331,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.681943127,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.681944820,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.681946525,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.681948255,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.681950815,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 2 2023-08-11T15:02:53.681952466,[INF]: subtest: 3 2023-08-11T15:02:53.681954058,[INF]: subcommand: 0 2023-08-11T15:02:53.681955660,[INF]: dimm: 0 2023-08-11T15:02:53.681957226,[INF]: mrank: 1 2023-08-11T15:02:53.681958821,[INF]: srank: 0 2023-08-11T15:02:53.681960426,[INF]: BG: 2 2023-08-11T15:02:53.681962017,[INF]: BA: 3 2023-08-11T15:02:53.681963674,[INF]: row: 0x0927a 2023-08-11T15:02:53.681965317,[INF]: col: 0x01 2023-08-11T15:02:53.681966934,[INF]: beat pair: 6/7 2023-08-11T15:02:53.681968515,[INF]: error mode: 2 2023-08-11T15:02:53.681970441,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.682051832,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.682054777,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.682056620,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.682058307,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.682060035,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.682061720,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.682064717,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 3 2023-08-11T15:02:53.682066417,[INF]: subtest: 3 2023-08-11T15:02:53.682068039,[INF]: subcommand: 0 2023-08-11T15:02:53.682069610,[INF]: dimm: 0 2023-08-11T15:02:53.682082382,[INF]: mrank: 1 2023-08-11T15:02:53.682085701,[INF]: srank: 0 2023-08-11T15:02:53.682087344,[INF]: BG: 3 2023-08-11T15:02:53.682088950,[INF]: BA: 1 2023-08-11T15:02:53.682090657,[INF]: row: 0x083f8 2023-08-11T15:02:53.682095034,[INF]: col: 0x12 2023-08-11T15:02:53.682096762,[INF]: beat pair: 6/7 2023-08-11T15:02:53.682098363,[INF]: error mode: 2 2023-08-11T15:02:53.682100598,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.682102176,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.682103781,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.682105484,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.682107165,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.682108869,[INF]: burst 7 actual : 0xdaaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.682110547,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.682113345,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 4 2023-08-11T15:02:53.682114996,[INF]: subtest: 3 2023-08-11T15:02:53.682116585,[INF]: subcommand: 0 2023-08-11T15:02:53.682118192,[INF]: dimm: 0 2023-08-11T15:02:53.682119758,[INF]: mrank: 1 2023-08-11T15:02:53.682121325,[INF]: srank: 0 2023-08-11T15:02:53.682122887,[INF]: BG: 2 2023-08-11T15:02:53.682124469,[INF]: BA: 3 2023-08-11T15:02:53.682126090,[INF]: row: 0x15703 2023-08-11T15:02:53.682127726,[INF]: col: 0x06 2023-08-11T15:02:53.682129340,[INF]: beat pair: 6/7 2023-08-11T15:02:53.682130905,[INF]: error mode: 2 2023-08-11T15:02:53.682132843,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.682134413,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.682135961,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.682137678,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.682139372,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.682141076,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.682142765,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.682145296,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 5 2023-08-11T15:02:53.682146931,[INF]: subtest: 3 2023-08-11T15:02:53.682148503,[INF]: subcommand: 0 2023-08-11T15:02:53.682150070,[INF]: dimm: 0 2023-08-11T15:02:53.682151641,[INF]: mrank: 0 2023-08-11T15:02:53.682153226,[INF]: srank: 0 2023-08-11T15:02:53.682154794,[INF]: BG: 2 2023-08-11T15:02:53.682156390,[INF]: BA: 0 2023-08-11T15:02:53.682157989,[INF]: row: 0x1e494 2023-08-11T15:02:53.682159613,[INF]: col: 0x36 2023-08-11T15:02:53.682161234,[INF]: beat pair: 6/7 2023-08-11T15:02:53.682162824,[INF]: error mode: 2 2023-08-11T15:02:53.682164746,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.682166309,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.682167895,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.682169599,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.682171308,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.682173029,[INF]: burst 7 actual : 0xbaaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.682174750,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.682177198,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 6 2023-08-11T15:02:53.682178826,[INF]: subtest: 3 2023-08-11T15:02:53.682180424,[INF]: subcommand: 0 2023-08-11T15:02:53.682181991,[INF]: dimm: 0 2023-08-11T15:02:53.682183584,[INF]: mrank: 1 2023-08-11T15:02:53.682185165,[INF]: srank: 0 2023-08-11T15:02:53.682186748,[INF]: BG: 1 2023-08-11T15:02:53.682188328,[INF]: BA: 3 2023-08-11T15:02:53.682189967,[INF]: row: 0x1417a 2023-08-11T15:02:53.682191560,[INF]: col: 0x3a 2023-08-11T15:02:53.682193169,[INF]: beat pair: 6/7 2023-08-11T15:02:53.682194741,[INF]: error mode: 2 2023-08-11T15:02:53.682196633,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.682198207,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.682199760,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.682201504,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.682203500,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.682205234,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.682206945,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.682209414,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 7 2023-08-11T15:02:53.682211054,[INF]: subtest: 3 2023-08-11T15:02:53.682212633,[INF]: subcommand: 0 2023-08-11T15:02:53.682214196,[INF]: dimm: 0 2023-08-11T15:02:53.682215794,[INF]: mrank: 1 2023-08-11T15:02:53.682217384,[INF]: srank: 0 2023-08-11T15:02:53.682218964,[INF]: BG: 2 2023-08-11T15:02:53.682220553,[INF]: BA: 3 2023-08-11T15:02:53.682222213,[INF]: row: 0x00920 2023-08-11T15:02:53.682223814,[INF]: col: 0x3b 2023-08-11T15:02:53.682225429,[INF]: beat pair: 6/7 2023-08-11T15:02:53.682227026,[INF]: error mode: 2 2023-08-11T15:02:53.682228936,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.682230495,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.682232064,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.682233771,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.682235484,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.682237171,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.682238854,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.682241304,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 8 2023-08-11T15:02:53.682242929,[INF]: subtest: 3 2023-08-11T15:02:53.682244508,[INF]: subcommand: 0 2023-08-11T15:02:53.682246082,[INF]: dimm: 0 2023-08-11T15:02:53.682247664,[INF]: mrank: 1 2023-08-11T15:02:53.682249253,[INF]: srank: 0 2023-08-11T15:02:53.682250833,[INF]: BG: 1 2023-08-11T15:02:53.682252416,[INF]: BA: 2 2023-08-11T15:02:53.682254020,[INF]: row: 0x12967 2023-08-11T15:02:53.682255636,[INF]: col: 0x04 2023-08-11T15:02:53.682257284,[INF]: beat pair: 6/7 2023-08-11T15:02:53.682258872,[INF]: error mode: 2 2023-08-11T15:02:53.682260768,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.682262349,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.682263920,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.682265644,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.682267333,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.682269022,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.682270730,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.682273172,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 9 2023-08-11T15:02:53.682274790,[INF]: subtest: 3 2023-08-11T15:02:53.682276385,[INF]: subcommand: 0 2023-08-11T15:02:53.682277959,[INF]: dimm: 0 2023-08-11T15:02:53.682279554,[INF]: mrank: 0 2023-08-11T15:02:53.682281124,[INF]: srank: 0 2023-08-11T15:02:53.682282716,[INF]: BG: 3 2023-08-11T15:02:53.682284282,[INF]: BA: 0 2023-08-11T15:02:53.682285896,[INF]: row: 0x02232 2023-08-11T15:02:53.682287515,[INF]: col: 0x16 2023-08-11T15:02:53.682289139,[INF]: beat pair: 6/7 2023-08-11T15:02:53.682290718,[INF]: error mode: 2 2023-08-11T15:02:53.682292609,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.682294158,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.682295725,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.682297433,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.682299142,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.682300849,[INF]: burst 7 actual : 0xbaaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.682302555,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.682305055,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 10 2023-08-11T15:02:53.682306681,[INF]: subtest: 3 2023-08-11T15:02:53.682308246,[INF]: subcommand: 0 2023-08-11T15:02:53.682310255,[INF]: dimm: 0 2023-08-11T15:02:53.682311853,[INF]: mrank: 1 2023-08-11T15:02:53.682313431,[INF]: srank: 0 2023-08-11T15:02:53.682315011,[INF]: BG: 1 2023-08-11T15:02:53.682316614,[INF]: BA: 1 2023-08-11T15:02:53.682318194,[INF]: row: 0x11d52 2023-08-11T15:02:53.682319780,[INF]: col: 0x27 2023-08-11T15:02:53.682321398,[INF]: beat pair: 6/7 2023-08-11T15:02:53.682322980,[INF]: error mode: 2 2023-08-11T15:02:53.682324874,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.682326437,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.682328004,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.682329722,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.682331412,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.682333137,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.682334843,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.682337308,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 11 2023-08-11T15:02:53.682338910,[INF]: subtest: 3 2023-08-11T15:02:53.682340479,[INF]: subcommand: 0 2023-08-11T15:02:53.682342067,[INF]: dimm: 0 2023-08-11T15:02:53.682343647,[INF]: mrank: 1 2023-08-11T15:02:53.682345236,[INF]: srank: 0 2023-08-11T15:02:53.682346804,[INF]: BG: 1 2023-08-11T15:02:53.682348376,[INF]: BA: 1 2023-08-11T15:02:53.682350007,[INF]: row: 0x00884 2023-08-11T15:02:53.682351601,[INF]: col: 0x3e 2023-08-11T15:02:53.682353225,[INF]: beat pair: 6/7 2023-08-11T15:02:53.682354827,[INF]: error mode: 2 2023-08-11T15:02:53.682356726,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.682358287,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.682359859,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.682656079,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.682659227,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.682660963,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.682662683,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.682666102,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 12 2023-08-11T15:02:53.682667804,[INF]: subtest: 3 2023-08-11T15:02:53.682669406,[INF]: subcommand: 0 2023-08-11T15:02:53.682671008,[INF]: dimm: 0 2023-08-11T15:02:53.682672586,[INF]: mrank: 1 2023-08-11T15:02:53.682674162,[INF]: srank: 0 2023-08-11T15:02:53.682675758,[INF]: BG: 3 2023-08-11T15:02:53.682677355,[INF]: BA: 1 2023-08-11T15:02:53.682679001,[INF]: row: 0x073bf 2023-08-11T15:02:53.682680619,[INF]: col: 0x0a 2023-08-11T15:02:53.682682280,[INF]: beat pair: 6/7 2023-08-11T15:02:53.682683883,[INF]: error mode: 2 2023-08-11T15:02:53.682686060,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.682687631,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.682689205,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.682690961,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.682692661,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.682694386,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.682696098,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.682698587,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 13 2023-08-11T15:02:53.682700231,[INF]: subtest: 3 2023-08-11T15:02:53.682701828,[INF]: subcommand: 0 2023-08-11T15:02:53.682703415,[INF]: dimm: 0 2023-08-11T15:02:53.682704995,[INF]: mrank: 1 2023-08-11T15:02:53.682706568,[INF]: srank: 0 2023-08-11T15:02:53.682708154,[INF]: BG: 1 2023-08-11T15:02:53.682709755,[INF]: BA: 0 2023-08-11T15:02:53.682711358,[INF]: row: 0x19a02 2023-08-11T15:02:53.682712984,[INF]: col: 0x01 2023-08-11T15:02:53.682714621,[INF]: beat pair: 6/7 2023-08-11T15:02:53.682716227,[INF]: error mode: 2 2023-08-11T15:02:53.682719446,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.682721052,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.682722640,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.682724345,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.682726045,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.682727786,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.682729484,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.682731967,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 14 2023-08-11T15:02:53.682733585,[INF]: subtest: 3 2023-08-11T15:02:53.682735170,[INF]: subcommand: 0 2023-08-11T15:02:53.682736746,[INF]: dimm: 0 2023-08-11T15:02:53.682738340,[INF]: mrank: 1 2023-08-11T15:02:53.682739935,[INF]: srank: 0 2023-08-11T15:02:53.682741499,[INF]: BG: 3 2023-08-11T15:02:53.682743090,[INF]: BA: 1 2023-08-11T15:02:53.682744687,[INF]: row: 0x17e50 2023-08-11T15:02:53.682746304,[INF]: col: 0x28 2023-08-11T15:02:53.682747940,[INF]: beat pair: 6/7 2023-08-11T15:02:53.682749528,[INF]: error mode: 2 2023-08-11T15:02:53.682751470,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.682753043,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.682754613,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.682756349,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.682835978,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.682838023,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.682839746,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.682842800,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 15 2023-08-11T15:02:53.682844509,[INF]: subtest: 3 2023-08-11T15:02:53.682846087,[INF]: subcommand: 0 2023-08-11T15:02:53.682847685,[INF]: dimm: 0 2023-08-11T15:02:53.682849258,[INF]: mrank: 1 2023-08-11T15:02:53.682850837,[INF]: srank: 0 2023-08-11T15:02:53.682852435,[INF]: BG: 3 2023-08-11T15:02:53.682854001,[INF]: BA: 2 2023-08-11T15:02:53.682855616,[INF]: row: 0x039d9 2023-08-11T15:02:53.682857229,[INF]: col: 0x28 2023-08-11T15:02:53.682858882,[INF]: beat pair: 6/7 2023-08-11T15:02:53.682860485,[INF]: error mode: 2 2023-08-11T15:02:53.682862480,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.682864076,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.682865664,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.682867344,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.682869036,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.682870736,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.682872437,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.682874871,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 16 2023-08-11T15:02:53.682876494,[INF]: subtest: 3 2023-08-11T15:02:53.682878089,[INF]: subcommand: 0 2023-08-11T15:02:53.682879660,[INF]: dimm: 0 2023-08-11T15:02:53.682881248,[INF]: mrank: 1 2023-08-11T15:02:53.682882823,[INF]: srank: 0 2023-08-11T15:02:53.682884386,[INF]: BG: 1 2023-08-11T15:02:53.682885985,[INF]: BA: 0 2023-08-11T15:02:53.682887611,[INF]: row: 0x07baf 2023-08-11T15:02:53.682889201,[INF]: col: 0x16 2023-08-11T15:02:53.682890829,[INF]: beat pair: 6/7 2023-08-11T15:02:53.682892417,[INF]: error mode: 2 2023-08-11T15:02:53.682894271,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.682895840,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.682897395,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.682899086,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.682900794,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.682902466,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.682905417,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.682907948,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 17 2023-08-11T15:02:53.682909574,[INF]: subtest: 3 2023-08-11T15:02:53.682911149,[INF]: subcommand: 0 2023-08-11T15:02:53.682912756,[INF]: dimm: 0 2023-08-11T15:02:53.682914348,[INF]: mrank: 1 2023-08-11T15:02:53.682915937,[INF]: srank: 0 2023-08-11T15:02:53.682917520,[INF]: BG: 3 2023-08-11T15:02:53.682919109,[INF]: BA: 0 2023-08-11T15:02:53.682920727,[INF]: row: 0x14970 2023-08-11T15:02:53.682922373,[INF]: col: 0x05 2023-08-11T15:02:53.682924016,[INF]: beat pair: 6/7 2023-08-11T15:02:53.682925605,[INF]: error mode: 2 2023-08-11T15:02:53.682927502,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.682929056,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.682930642,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.682932340,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.682934027,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.682935709,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.682937399,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.682939848,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 18 2023-08-11T15:02:53.682941491,[INF]: subtest: 3 2023-08-11T15:02:53.682943057,[INF]: subcommand: 0 2023-08-11T15:02:53.682944660,[INF]: dimm: 0 2023-08-11T15:02:53.682946237,[INF]: mrank: 1 2023-08-11T15:02:53.682947824,[INF]: srank: 0 2023-08-11T15:02:53.682949385,[INF]: BG: 3 2023-08-11T15:02:53.682950962,[INF]: BA: 1 2023-08-11T15:02:53.682952551,[INF]: row: 0x12c30 2023-08-11T15:02:53.682954131,[INF]: col: 0x35 2023-08-11T15:02:53.682955777,[INF]: beat pair: 6/7 2023-08-11T15:02:53.682957352,[INF]: error mode: 2 2023-08-11T15:02:53.682959213,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.682960781,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.682962362,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.682964047,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.682965742,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.682967410,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.682969116,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.682971544,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 19 2023-08-11T15:02:53.682973157,[INF]: subtest: 3 2023-08-11T15:02:53.682974755,[INF]: subcommand: 0 2023-08-11T15:02:53.682976334,[INF]: dimm: 0 2023-08-11T15:02:53.682977927,[INF]: mrank: 1 2023-08-11T15:02:53.682979506,[INF]: srank: 0 2023-08-11T15:02:53.682981081,[INF]: BG: 0 2023-08-11T15:02:53.682982666,[INF]: BA: 0 2023-08-11T15:02:53.682984280,[INF]: row: 0x05467 2023-08-11T15:02:53.682985846,[INF]: col: 0x13 2023-08-11T15:02:53.682987460,[INF]: beat pair: 6/7 2023-08-11T15:02:53.682989050,[INF]: error mode: 2 2023-08-11T15:02:53.682990917,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.682992463,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.682994029,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.682995713,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.682997421,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.682999098,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.683000793,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.683003231,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 20 2023-08-11T15:02:53.683004848,[INF]: subtest: 3 2023-08-11T15:02:53.683006429,[INF]: subcommand: 0 2023-08-11T15:02:53.683007996,[INF]: dimm: 0 2023-08-11T15:02:53.683009567,[INF]: mrank: 1 2023-08-11T15:02:53.683011153,[INF]: srank: 0 2023-08-11T15:02:53.683013184,[INF]: BG: 3 2023-08-11T15:02:53.683014794,[INF]: BA: 0 2023-08-11T15:02:53.683016395,[INF]: row: 0x0de6c 2023-08-11T15:02:53.683017986,[INF]: col: 0x3b 2023-08-11T15:02:53.683019605,[INF]: beat pair: 6/7 2023-08-11T15:02:53.683021198,[INF]: error mode: 2 2023-08-11T15:02:53.683023067,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.683024610,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.683026186,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.683027870,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.683029554,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.683031214,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.683032950,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.683035410,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 21 2023-08-11T15:02:53.683037040,[INF]: subtest: 3 2023-08-11T15:02:53.683038612,[INF]: subcommand: 0 2023-08-11T15:02:53.683040179,[INF]: dimm: 0 2023-08-11T15:02:53.683041764,[INF]: mrank: 1 2023-08-11T15:02:53.683043354,[INF]: srank: 0 2023-08-11T15:02:53.683044938,[INF]: BG: 3 2023-08-11T15:02:53.683046536,[INF]: BA: 0 2023-08-11T15:02:53.683048140,[INF]: row: 0x01a7a 2023-08-11T15:02:53.683049744,[INF]: col: 0x0d 2023-08-11T15:02:53.683051363,[INF]: beat pair: 6/7 2023-08-11T15:02:53.683052950,[INF]: error mode: 2 2023-08-11T15:02:53.683054809,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.683056373,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.683057950,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.683059617,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.683061301,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.683062978,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.683075964,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.683080503,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 22 2023-08-11T15:02:53.683083506,[INF]: subtest: 3 2023-08-11T15:02:53.683086312,[INF]: subcommand: 0 2023-08-11T15:02:53.683088846,[INF]: dimm: 0 2023-08-11T15:02:53.683091287,[INF]: mrank: 1 2023-08-11T15:02:53.683094072,[INF]: srank: 0 2023-08-11T15:02:53.683096880,[INF]: BG: 2 2023-08-11T15:02:53.683173937,[INF]: BA: 1 2023-08-11T15:02:53.683178846,[INF]: row: 0x0a133 2023-08-11T15:02:53.683180518,[INF]: col: 0x0c 2023-08-11T15:02:53.683182148,[INF]: beat pair: 6/7 2023-08-11T15:02:53.683183745,[INF]: error mode: 2 2023-08-11T15:02:53.683186056,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.683187631,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.683189229,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.683190961,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.683192658,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.683194354,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.683196041,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.683199031,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 23 2023-08-11T15:02:53.683200704,[INF]: subtest: 3 2023-08-11T15:02:53.683202285,[INF]: subcommand: 0 2023-08-11T15:02:53.683203864,[INF]: dimm: 0 2023-08-11T15:02:53.683205466,[INF]: mrank: 1 2023-08-11T15:02:53.683207052,[INF]: srank: 0 2023-08-11T15:02:53.683208642,[INF]: BG: 1 2023-08-11T15:02:53.683210204,[INF]: BA: 0 2023-08-11T15:02:53.683211831,[INF]: row: 0x1628f 2023-08-11T15:02:53.683213443,[INF]: col: 0x2d 2023-08-11T15:02:53.683215062,[INF]: beat pair: 6/7 2023-08-11T15:02:53.683216636,[INF]: error mode: 2 2023-08-11T15:02:53.683218492,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.683220084,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.683223104,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.683224829,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.683226535,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.683228248,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.683229970,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.683232494,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 24 2023-08-11T15:02:53.683234132,[INF]: subtest: 3 2023-08-11T15:02:53.683235724,[INF]: subcommand: 0 2023-08-11T15:02:53.683237318,[INF]: dimm: 0 2023-08-11T15:02:53.683238900,[INF]: mrank: 1 2023-08-11T15:02:53.683240512,[INF]: srank: 0 2023-08-11T15:02:53.683242096,[INF]: BG: 3 2023-08-11T15:02:53.683243685,[INF]: BA: 2 2023-08-11T15:02:53.683245301,[INF]: row: 0x177bb 2023-08-11T15:02:53.683246904,[INF]: col: 0x37 2023-08-11T15:02:53.683248526,[INF]: beat pair: 6/7 2023-08-11T15:02:53.683250123,[INF]: error mode: 2 2023-08-11T15:02:53.683252016,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.683253580,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.683255159,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.683256878,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.683258598,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.683260311,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.683262024,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.683264500,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 25 2023-08-11T15:02:53.683266135,[INF]: subtest: 3 2023-08-11T15:02:53.683267711,[INF]: subcommand: 0 2023-08-11T15:02:53.683269318,[INF]: dimm: 0 2023-08-11T15:02:53.683270914,[INF]: mrank: 1 2023-08-11T15:02:53.683272512,[INF]: srank: 0 2023-08-11T15:02:53.683274089,[INF]: BG: 1 2023-08-11T15:02:53.683275685,[INF]: BA: 2 2023-08-11T15:02:53.683277289,[INF]: row: 0x0fcd6 2023-08-11T15:02:53.683278895,[INF]: col: 0x15 2023-08-11T15:02:53.683280515,[INF]: beat pair: 6/7 2023-08-11T15:02:53.683282093,[INF]: error mode: 2 2023-08-11T15:02:53.683283980,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.683285547,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.683287138,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.683288854,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.683290557,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.683292274,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.683293987,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.683296512,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 26 2023-08-11T15:02:53.683298125,[INF]: subtest: 3 2023-08-11T15:02:53.683299719,[INF]: subcommand: 0 2023-08-11T15:02:53.683301304,[INF]: dimm: 0 2023-08-11T15:02:53.683302914,[INF]: mrank: 1 2023-08-11T15:02:53.683304497,[INF]: srank: 0 2023-08-11T15:02:53.683306095,[INF]: BG: 1 2023-08-11T15:02:53.683307680,[INF]: BA: 2 2023-08-11T15:02:53.683309297,[INF]: row: 0x1eee5 2023-08-11T15:02:53.683310905,[INF]: col: 0x19 2023-08-11T15:02:53.683312548,[INF]: beat pair: 6/7 2023-08-11T15:02:53.683314138,[INF]: error mode: 2 2023-08-11T15:02:53.683316024,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.683317610,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.683319192,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.683320893,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.683322582,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.683324303,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.683326018,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.683328477,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 27 2023-08-11T15:02:53.683330563,[INF]: subtest: 3 2023-08-11T15:02:53.683332150,[INF]: subcommand: 0 2023-08-11T15:02:53.683333739,[INF]: dimm: 0 2023-08-11T15:02:53.683335324,[INF]: mrank: 1 2023-08-11T15:02:53.683336913,[INF]: srank: 0 2023-08-11T15:02:53.683338494,[INF]: BG: 3 2023-08-11T15:02:53.683340075,[INF]: BA: 2 2023-08-11T15:02:53.683341709,[INF]: row: 0x0dd49 2023-08-11T15:02:53.683343345,[INF]: col: 0x2e 2023-08-11T15:02:53.683344973,[INF]: beat pair: 6/7 2023-08-11T15:02:53.683346564,[INF]: error mode: 2 2023-08-11T15:02:53.683348538,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.683350130,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.683351683,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.683353385,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.683355099,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.683356808,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.683358518,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.683360991,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 28 2023-08-11T15:02:53.683362609,[INF]: subtest: 3 2023-08-11T15:02:53.683364210,[INF]: subcommand: 0 2023-08-11T15:02:53.683365795,[INF]: dimm: 0 2023-08-11T15:02:53.683367381,[INF]: mrank: 1 2023-08-11T15:02:53.683368963,[INF]: srank: 0 2023-08-11T15:02:53.683370555,[INF]: BG: 3 2023-08-11T15:02:53.683372130,[INF]: BA: 0 2023-08-11T15:02:53.683373747,[INF]: row: 0x051ba 2023-08-11T15:02:53.683375344,[INF]: col: 0x1b 2023-08-11T15:02:53.683376961,[INF]: beat pair: 6/7 2023-08-11T15:02:53.683378538,[INF]: error mode: 2 2023-08-11T15:02:53.683380436,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.683382018,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.683383587,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.683385293,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.683386988,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.683388696,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.683390417,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.683392885,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 29 2023-08-11T15:02:53.683394515,[INF]: subtest: 3 2023-08-11T15:02:53.683396100,[INF]: subcommand: 0 2023-08-11T15:02:53.683397687,[INF]: dimm: 0 2023-08-11T15:02:53.683399273,[INF]: mrank: 1 2023-08-11T15:02:53.683400888,[INF]: srank: 0 2023-08-11T15:02:53.683402490,[INF]: BG: 2 2023-08-11T15:02:53.683404084,[INF]: BA: 0 2023-08-11T15:02:53.683405677,[INF]: row: 0x18664 2023-08-11T15:02:53.683407321,[INF]: col: 0x02 2023-08-11T15:02:53.683408968,[INF]: beat pair: 6/7 2023-08-11T15:02:53.683410557,[INF]: error mode: 2 2023-08-11T15:02:53.683412450,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.683414041,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.683415603,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.683417303,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.683419013,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.683420732,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.683422440,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.683424887,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 30 2023-08-11T15:02:53.683426513,[INF]: subtest: 3 2023-08-11T15:02:53.683428109,[INF]: subcommand: 0 2023-08-11T15:02:53.683429708,[INF]: dimm: 0 2023-08-11T15:02:53.683431288,[INF]: mrank: 1 2023-08-11T15:02:53.683432909,[INF]: srank: 0 2023-08-11T15:02:53.683434486,[INF]: BG: 1 2023-08-11T15:02:53.683436071,[INF]: BA: 2 2023-08-11T15:02:53.683437688,[INF]: row: 0x0e97e 2023-08-11T15:02:53.683439939,[INF]: col: 0x38 2023-08-11T15:02:53.683441617,[INF]: beat pair: 6/7 2023-08-11T15:02:53.683443206,[INF]: error mode: 2 2023-08-11T15:02:53.683445122,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.683446697,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.683448266,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.683449980,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.683451691,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.683453396,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.683455104,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.683457569,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 31 2023-08-11T15:02:53.683459205,[INF]: subtest: 3 2023-08-11T15:02:53.683460792,[INF]: subcommand: 0 2023-08-11T15:02:53.683462402,[INF]: dimm: 0 2023-08-11T15:02:53.683463996,[INF]: mrank: 1 2023-08-11T15:02:53.683465591,[INF]: srank: 0 2023-08-11T15:02:53.683467196,[INF]: BG: 2 2023-08-11T15:02:53.683468784,[INF]: BA: 2 2023-08-11T15:02:53.683470372,[INF]: row: 0x11072 2023-08-11T15:02:53.683472006,[INF]: col: 0x06 2023-08-11T15:02:53.683473633,[INF]: beat pair: 6/7 2023-08-11T15:02:53.683475234,[INF]: error mode: 2 2023-08-11T15:02:53.683773687,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.683778822,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.683780625,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.683782415,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.683784161,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.683785885,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.683787559,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.683790776,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 32 2023-08-11T15:02:53.683792466,[INF]: subtest: 3 2023-08-11T15:02:53.683794068,[INF]: subcommand: 0 2023-08-11T15:02:53.683795639,[INF]: dimm: 0 2023-08-11T15:02:53.683797210,[INF]: mrank: 1 2023-08-11T15:02:53.683798777,[INF]: srank: 0 2023-08-11T15:02:53.683800367,[INF]: BG: 3 2023-08-11T15:02:53.683801963,[INF]: BA: 3 2023-08-11T15:02:53.683803550,[INF]: row: 0x1bed3 2023-08-11T15:02:53.683805151,[INF]: col: 0x24 2023-08-11T15:02:53.683806803,[INF]: beat pair: 6/7 2023-08-11T15:02:53.683808385,[INF]: error mode: 2 2023-08-11T15:02:53.683810444,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.683812013,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.683813627,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.683815330,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.683817026,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.683818703,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.683820394,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.683822807,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 33 2023-08-11T15:02:53.683824450,[INF]: subtest: 3 2023-08-11T15:02:53.683826010,[INF]: subcommand: 0 2023-08-11T15:02:53.683827569,[INF]: dimm: 0 2023-08-11T15:02:53.683829142,[INF]: mrank: 1 2023-08-11T15:02:53.683830749,[INF]: srank: 0 2023-08-11T15:02:53.683832304,[INF]: BG: 3 2023-08-11T15:02:53.683833926,[INF]: BA: 3 2023-08-11T15:02:53.683835527,[INF]: row: 0x0752c 2023-08-11T15:02:53.683837113,[INF]: col: 0x2b 2023-08-11T15:02:53.683838738,[INF]: beat pair: 6/7 2023-08-11T15:02:53.683840314,[INF]: error mode: 2 2023-08-11T15:02:53.683842169,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.683843725,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.683845309,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.683846986,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.683849889,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.683851641,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.683853350,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.683855827,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 34 2023-08-11T15:02:53.683857425,[INF]: subtest: 3 2023-08-11T15:02:53.683859014,[INF]: subcommand: 0 2023-08-11T15:02:53.683860601,[INF]: dimm: 0 2023-08-11T15:02:53.683862162,[INF]: mrank: 1 2023-08-11T15:02:53.683863766,[INF]: srank: 0 2023-08-11T15:02:53.683941580,[INF]: BG: 1 2023-08-11T15:02:53.683943530,[INF]: BA: 2 2023-08-11T15:02:53.683945208,[INF]: row: 0x00cd7 2023-08-11T15:02:53.683946809,[INF]: col: 0x39 2023-08-11T15:02:53.683948458,[INF]: beat pair: 6/7 2023-08-11T15:02:53.683950057,[INF]: error mode: 2 2023-08-11T15:02:53.683952284,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.683953910,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.683955470,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.683957192,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.683958892,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.683960635,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.683962324,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.683965080,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 35 2023-08-11T15:02:53.683966705,[INF]: subtest: 3 2023-08-11T15:02:53.683968289,[INF]: subcommand: 0 2023-08-11T15:02:53.683969891,[INF]: dimm: 0 2023-08-11T15:02:53.683971466,[INF]: mrank: 1 2023-08-11T15:02:53.683973037,[INF]: srank: 0 2023-08-11T15:02:53.683974615,[INF]: BG: 2 2023-08-11T15:02:53.683976176,[INF]: BA: 3 2023-08-11T15:02:53.683977787,[INF]: row: 0x11d3e 2023-08-11T15:02:53.683979362,[INF]: col: 0x23 2023-08-11T15:02:53.683980957,[INF]: beat pair: 6/7 2023-08-11T15:02:53.683982525,[INF]: error mode: 2 2023-08-11T15:02:53.683984433,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.683986004,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.683987579,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.683989296,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.683990988,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.683992712,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.683994430,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.683996875,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 36 2023-08-11T15:02:53.683998503,[INF]: subtest: 3 2023-08-11T15:02:53.684000091,[INF]: subcommand: 0 2023-08-11T15:02:53.684001679,[INF]: dimm: 0 2023-08-11T15:02:53.684003240,[INF]: mrank: 1 2023-08-11T15:02:53.684004823,[INF]: srank: 0 2023-08-11T15:02:53.684006413,[INF]: BG: 3 2023-08-11T15:02:53.684007975,[INF]: BA: 2 2023-08-11T15:02:53.684009574,[INF]: row: 0x08fa6 2023-08-11T15:02:53.684011166,[INF]: col: 0x34 2023-08-11T15:02:53.684012762,[INF]: beat pair: 6/7 2023-08-11T15:02:53.684014333,[INF]: error mode: 2 2023-08-11T15:02:53.684016194,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.684017807,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.684019370,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.684021066,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.684022765,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.684024476,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.684026175,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.684028625,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 37 2023-08-11T15:02:53.684030235,[INF]: subtest: 3 2023-08-11T15:02:53.684031806,[INF]: subcommand: 0 2023-08-11T15:02:53.684034647,[INF]: dimm: 0 2023-08-11T15:02:53.684036305,[INF]: mrank: 1 2023-08-11T15:02:53.684037888,[INF]: srank: 0 2023-08-11T15:02:53.684039465,[INF]: BG: 1 2023-08-11T15:02:53.684041051,[INF]: BA: 1 2023-08-11T15:02:53.684042665,[INF]: row: 0x1344d 2023-08-11T15:02:53.684044263,[INF]: col: 0x38 2023-08-11T15:02:53.684045877,[INF]: beat pair: 6/7 2023-08-11T15:02:53.684047448,[INF]: error mode: 2 2023-08-11T15:02:53.684049350,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.684050907,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.684052466,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.684054198,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.684067816,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.684069816,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.684071538,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.684074485,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 38 2023-08-11T15:02:53.684076138,[INF]: subtest: 3 2023-08-11T15:02:53.684077747,[INF]: subcommand: 0 2023-08-11T15:02:53.684079329,[INF]: dimm: 0 2023-08-11T15:02:53.684080876,[INF]: mrank: 1 2023-08-11T15:02:53.684082474,[INF]: srank: 0 2023-08-11T15:02:53.684084040,[INF]: BG: 1 2023-08-11T15:02:53.684085606,[INF]: BA: 0 2023-08-11T15:02:53.684087213,[INF]: row: 0x15dce 2023-08-11T15:02:53.684088815,[INF]: col: 0x22 2023-08-11T15:02:53.684090429,[INF]: beat pair: 6/7 2023-08-11T15:02:53.684092010,[INF]: error mode: 2 2023-08-11T15:02:53.684094023,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.684095629,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.684097194,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.684098873,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.684100575,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.684102254,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.684103953,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.684106393,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 39 2023-08-11T15:02:53.684108021,[INF]: subtest: 3 2023-08-11T15:02:53.684109604,[INF]: subcommand: 0 2023-08-11T15:02:53.684111148,[INF]: dimm: 0 2023-08-11T15:02:53.684112741,[INF]: mrank: 1 2023-08-11T15:02:53.684114324,[INF]: srank: 0 2023-08-11T15:02:53.684115884,[INF]: BG: 2 2023-08-11T15:02:53.684117459,[INF]: BA: 3 2023-08-11T15:02:53.684119050,[INF]: row: 0x00aeb 2023-08-11T15:02:53.684120657,[INF]: col: 0x06 2023-08-11T15:02:53.684122269,[INF]: beat pair: 6/7 2023-08-11T15:02:53.684123883,[INF]: error mode: 2 2023-08-11T15:02:53.684125731,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.684127286,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.684128874,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.684130568,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.684132259,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.684133935,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.684135615,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.684138034,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 40 2023-08-11T15:02:53.684139664,[INF]: subtest: 3 2023-08-11T15:02:53.684141235,[INF]: subcommand: 0 2023-08-11T15:02:53.684142800,[INF]: dimm: 0 2023-08-11T15:02:53.684144381,[INF]: mrank: 1 2023-08-11T15:02:53.684145938,[INF]: srank: 0 2023-08-11T15:02:53.684147503,[INF]: BG: 0 2023-08-11T15:02:53.684149075,[INF]: BA: 3 2023-08-11T15:02:53.684150675,[INF]: row: 0x173cb 2023-08-11T15:02:53.684152233,[INF]: col: 0x1e 2023-08-11T15:02:53.684153844,[INF]: beat pair: 6/7 2023-08-11T15:02:53.684155421,[INF]: error mode: 2 2023-08-11T15:02:53.684158415,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.684160088,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.684161669,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.684163356,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.684165047,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.684166735,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.684168438,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.684170872,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 41 2023-08-11T15:02:53.684172465,[INF]: subtest: 3 2023-08-11T15:02:53.684174045,[INF]: subcommand: 0 2023-08-11T15:02:53.684175594,[INF]: dimm: 0 2023-08-11T15:02:53.684177154,[INF]: mrank: 1 2023-08-11T15:02:53.684178745,[INF]: srank: 0 2023-08-11T15:02:53.684180295,[INF]: BG: 2 2023-08-11T15:02:53.684181874,[INF]: BA: 2 2023-08-11T15:02:53.684183454,[INF]: row: 0x1f3c6 2023-08-11T15:02:53.684185039,[INF]: col: 0x11 2023-08-11T15:02:53.684186635,[INF]: beat pair: 6/7 2023-08-11T15:02:53.684188198,[INF]: error mode: 2 2023-08-11T15:02:53.684190075,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.684191637,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.684193212,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.684194914,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.684196599,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.684198285,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.684199984,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.684202422,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 42 2023-08-11T15:02:53.684204044,[INF]: subtest: 3 2023-08-11T15:02:53.684205608,[INF]: subcommand: 0 2023-08-11T15:02:53.684207177,[INF]: dimm: 0 2023-08-11T15:02:53.684208743,[INF]: mrank: 1 2023-08-11T15:02:53.684210316,[INF]: srank: 0 2023-08-11T15:02:53.684211889,[INF]: BG: 0 2023-08-11T15:02:53.684213459,[INF]: BA: 2 2023-08-11T15:02:53.684215054,[INF]: row: 0x18ffd 2023-08-11T15:02:53.684216664,[INF]: col: 0x0f 2023-08-11T15:02:53.684218270,[INF]: beat pair: 6/7 2023-08-11T15:02:53.684536092,[INF]: error mode: 2 2023-08-11T15:02:53.684541924,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.684543608,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.684545223,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.684547062,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.684548768,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.684550518,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.684569249,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.684574532,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 43 2023-08-11T15:02:53.684576270,[INF]: subtest: 3 2023-08-11T15:02:53.684577910,[INF]: subcommand: 0 2023-08-11T15:02:53.684579519,[INF]: dimm: 0 2023-08-11T15:02:53.684581120,[INF]: mrank: 1 2023-08-11T15:02:53.684582743,[INF]: srank: 0 2023-08-11T15:02:53.684584346,[INF]: BG: 1 2023-08-11T15:02:53.684585986,[INF]: BA: 0 2023-08-11T15:02:53.684587597,[INF]: row: 0x1b886 2023-08-11T15:02:53.684589231,[INF]: col: 0x19 2023-08-11T15:02:53.684590891,[INF]: beat pair: 6/7 2023-08-11T15:02:53.684592498,[INF]: error mode: 2 2023-08-11T15:02:53.684594632,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.684596227,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.684597835,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.684599578,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.684601294,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.684603039,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.684606447,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.684609021,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 44 2023-08-11T15:02:53.684610646,[INF]: subtest: 3 2023-08-11T15:02:53.684612241,[INF]: subcommand: 0 2023-08-11T15:02:53.684613838,[INF]: dimm: 0 2023-08-11T15:02:53.684615462,[INF]: mrank: 1 2023-08-11T15:02:53.684617050,[INF]: srank: 0 2023-08-11T15:02:53.684618660,[INF]: BG: 2 2023-08-11T15:02:53.684620271,[INF]: BA: 3 2023-08-11T15:02:53.684697369,[INF]: row: 0x1f4c0 2023-08-11T15:02:53.684700002,[INF]: col: 0x0e 2023-08-11T15:02:53.684701677,[INF]: beat pair: 6/7 2023-08-11T15:02:53.684703266,[INF]: error mode: 2 2023-08-11T15:02:53.684705482,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.684707063,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.684708646,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.684710401,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.684712121,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.684713821,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.684715517,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.684718272,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 45 2023-08-11T15:02:53.684719976,[INF]: subtest: 3 2023-08-11T15:02:53.684721568,[INF]: subcommand: 0 2023-08-11T15:02:53.684723164,[INF]: dimm: 0 2023-08-11T15:02:53.684724742,[INF]: mrank: 1 2023-08-11T15:02:53.684726333,[INF]: srank: 0 2023-08-11T15:02:53.684727914,[INF]: BG: 2 2023-08-11T15:02:53.684729500,[INF]: BA: 0 2023-08-11T15:02:53.684731180,[INF]: row: 0x0f85c 2023-08-11T15:02:53.684732769,[INF]: col: 0x17 2023-08-11T15:02:53.684734437,[INF]: beat pair: 6/7 2023-08-11T15:02:53.684736023,[INF]: error mode: 2 2023-08-11T15:02:53.684737933,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.684739526,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.684741101,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.684742837,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.684744530,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.684746243,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.684747942,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.684750437,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 46 2023-08-11T15:02:53.684752071,[INF]: subtest: 3 2023-08-11T15:02:53.684753677,[INF]: subcommand: 0 2023-08-11T15:02:53.684755269,[INF]: dimm: 0 2023-08-11T15:02:53.684756855,[INF]: mrank: 1 2023-08-11T15:02:53.684758421,[INF]: srank: 0 2023-08-11T15:02:53.684759992,[INF]: BG: 1 2023-08-11T15:02:53.684761589,[INF]: BA: 1 2023-08-11T15:02:53.684763197,[INF]: row: 0x1bb41 2023-08-11T15:02:53.684764822,[INF]: col: 0x02 2023-08-11T15:02:53.684766449,[INF]: beat pair: 6/7 2023-08-11T15:02:53.684768041,[INF]: error mode: 2 2023-08-11T15:02:53.684769947,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.684771509,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.684773077,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.684774797,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.684776506,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.684778226,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.684779920,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.684782449,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 47 2023-08-11T15:02:53.684784094,[INF]: subtest: 3 2023-08-11T15:02:53.684785697,[INF]: subcommand: 0 2023-08-11T15:02:53.684787280,[INF]: dimm: 0 2023-08-11T15:02:53.684788861,[INF]: mrank: 1 2023-08-11T15:02:53.684790447,[INF]: srank: 0 2023-08-11T15:02:53.684793182,[INF]: BG: 2 2023-08-11T15:02:53.684794814,[INF]: BA: 0 2023-08-11T15:02:53.684796420,[INF]: row: 0x19d5c 2023-08-11T15:02:53.684798024,[INF]: col: 0x24 2023-08-11T15:02:53.684799645,[INF]: beat pair: 6/7 2023-08-11T15:02:53.684801230,[INF]: error mode: 2 2023-08-11T15:02:53.684803117,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.684804700,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.684806281,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.684807980,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.684809668,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.684811354,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.684813060,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.684815521,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 48 2023-08-11T15:02:53.684817153,[INF]: subtest: 3 2023-08-11T15:02:53.684818759,[INF]: subcommand: 0 2023-08-11T15:02:53.684820347,[INF]: dimm: 0 2023-08-11T15:02:53.684821925,[INF]: mrank: 1 2023-08-11T15:02:53.684823510,[INF]: srank: 0 2023-08-11T15:02:53.684825070,[INF]: BG: 2 2023-08-11T15:02:53.684826640,[INF]: BA: 1 2023-08-11T15:02:53.684828247,[INF]: row: 0x13f9d 2023-08-11T15:02:53.684829823,[INF]: col: 0x3f 2023-08-11T15:02:53.684831441,[INF]: beat pair: 6/7 2023-08-11T15:02:53.684833042,[INF]: error mode: 2 2023-08-11T15:02:53.684834925,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.684836508,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.684838071,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.684839760,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.684841460,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.684843159,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.684844852,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.684847281,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 49 2023-08-11T15:02:53.684848922,[INF]: subtest: 3 2023-08-11T15:02:53.684850516,[INF]: subcommand: 0 2023-08-11T15:02:53.684852073,[INF]: dimm: 0 2023-08-11T15:02:53.684853656,[INF]: mrank: 1 2023-08-11T15:02:53.684855231,[INF]: srank: 0 2023-08-11T15:02:53.684856794,[INF]: BG: 0 2023-08-11T15:02:53.684858405,[INF]: BA: 2 2023-08-11T15:02:53.684860007,[INF]: row: 0x1c84b 2023-08-11T15:02:53.684861617,[INF]: col: 0x02 2023-08-11T15:02:53.684863246,[INF]: beat pair: 6/7 2023-08-11T15:02:53.684864848,[INF]: error mode: 2 2023-08-11T15:02:53.684866731,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.684868316,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.684869879,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.684871589,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.684873320,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.684875014,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.684876711,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.684879152,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 50 2023-08-11T15:02:53.684880803,[INF]: subtest: 3 2023-08-11T15:02:53.684882394,[INF]: subcommand: 0 2023-08-11T15:02:53.684883965,[INF]: dimm: 0 2023-08-11T15:02:53.684885550,[INF]: mrank: 1 2023-08-11T15:02:53.684887145,[INF]: srank: 0 2023-08-11T15:02:53.684888720,[INF]: BG: 0 2023-08-11T15:02:53.684890297,[INF]: BA: 0 2023-08-11T15:02:53.684891930,[INF]: row: 0x158b1 2023-08-11T15:02:53.684893522,[INF]: col: 0x3d 2023-08-11T15:02:53.684895145,[INF]: beat pair: 6/7 2023-08-11T15:02:53.684896747,[INF]: error mode: 2 2023-08-11T15:02:53.684898624,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.684900212,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.684902361,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.684904069,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.684905788,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.684907475,[INF]: burst 7 actual : 0x1aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.684909157,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.684911600,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 51 2023-08-11T15:02:53.684913258,[INF]: subtest: 3 2023-08-11T15:02:53.684914844,[INF]: subcommand: 0 2023-08-11T15:02:53.684916410,[INF]: dimm: 0 2023-08-11T15:02:53.684917983,[INF]: mrank: 1 2023-08-11T15:02:53.684919580,[INF]: srank: 0 2023-08-11T15:02:53.684921129,[INF]: BG: 2 2023-08-11T15:02:53.684922701,[INF]: BA: 1 2023-08-11T15:02:53.684924297,[INF]: row: 0x1e65a 2023-08-11T15:02:53.684925932,[INF]: col: 0x0d 2023-08-11T15:02:53.684927556,[INF]: beat pair: 6/7 2023-08-11T15:02:53.684929139,[INF]: error mode: 2 2023-08-11T15:02:53.684931035,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.684932626,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.684934196,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.684935883,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.684937595,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.684939292,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.684941021,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.684943447,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 52 2023-08-11T15:02:53.684945084,[INF]: subtest: 3 2023-08-11T15:02:53.684946662,[INF]: subcommand: 0 2023-08-11T15:02:53.684948212,[INF]: dimm: 0 2023-08-11T15:02:53.684949804,[INF]: mrank: 1 2023-08-11T15:02:53.684951386,[INF]: srank: 0 2023-08-11T15:02:53.684952945,[INF]: BG: 2 2023-08-11T15:02:53.684954534,[INF]: BA: 2 2023-08-11T15:02:53.684956160,[INF]: row: 0x09bc4 2023-08-11T15:02:53.684957777,[INF]: col: 0x00 2023-08-11T15:02:53.684959392,[INF]: beat pair: 6/7 2023-08-11T15:02:53.684960979,[INF]: error mode: 2 2023-08-11T15:02:53.684962855,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.684964448,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.684966041,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.684967735,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.684969434,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.684971161,[INF]: burst 7 actual : 0xdaaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.684972853,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.684975365,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 53 2023-08-11T15:02:53.684977028,[INF]: subtest: 3 2023-08-11T15:02:53.684978597,[INF]: subcommand: 0 2023-08-11T15:02:53.684980197,[INF]: dimm: 0 2023-08-11T15:02:53.684981759,[INF]: mrank: 1 2023-08-11T15:02:53.685279798,[INF]: srank: 0 2023-08-11T15:02:53.685282893,[INF]: BG: 3 2023-08-11T15:02:53.685284552,[INF]: BA: 0 2023-08-11T15:02:53.685286194,[INF]: row: 0x1c86b 2023-08-11T15:02:53.685287784,[INF]: col: 0x35 2023-08-11T15:02:53.685289435,[INF]: beat pair: 6/7 2023-08-11T15:02:53.685291017,[INF]: error mode: 2 2023-08-11T15:02:53.685293352,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.685294934,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.685296514,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.685298231,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.685299932,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.685301633,[INF]: burst 7 actual : 0x2aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.685303320,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.685306149,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 54 2023-08-11T15:02:53.686089897,[INF]: subtest: 3 2023-08-11T15:02:53.686096552,[INF]: subcommand: 0 2023-08-11T15:02:53.686098303,[INF]: dimm: 0 2023-08-11T15:02:53.686099924,[INF]: mrank: 1 2023-08-11T15:02:53.686101522,[INF]: srank: 0 2023-08-11T15:02:53.686103098,[INF]: BG: 3 2023-08-11T15:02:53.686104683,[INF]: BA: 1 2023-08-11T15:02:53.686106448,[INF]: row: 0x1e2ec 2023-08-11T15:02:53.686108268,[INF]: col: 0x07 2023-08-11T15:02:53.686109925,[INF]: beat pair: 6/7 2023-08-11T15:02:53.686111534,[INF]: error mode: 2 2023-08-11T15:02:53.686114499,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.686116163,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.686117807,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.686119622,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.686121399,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.686123135,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.686124842,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.686128310,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 55 2023-08-11T15:02:53.686129993,[INF]: subtest: 3 2023-08-11T15:02:53.686131607,[INF]: subcommand: 0 2023-08-11T15:02:53.686133204,[INF]: dimm: 0 2023-08-11T15:02:53.686134755,[INF]: mrank: 1 2023-08-11T15:02:53.686136334,[INF]: srank: 0 2023-08-11T15:02:53.686137920,[INF]: BG: 3 2023-08-11T15:02:53.686139525,[INF]: BA: 1 2023-08-11T15:02:53.686141199,[INF]: row: 0x0b5ad 2023-08-11T15:02:53.686142828,[INF]: col: 0x2d 2023-08-11T15:02:53.686144450,[INF]: beat pair: 6/7 2023-08-11T15:02:53.686146047,[INF]: error mode: 2 2023-08-11T15:02:53.686148092,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.686149660,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.686151259,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.686152979,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.686154699,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.686156413,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.686158119,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.686160611,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 56 2023-08-11T15:02:53.686162235,[INF]: subtest: 3 2023-08-11T15:02:53.686163822,[INF]: subcommand: 0 2023-08-11T15:02:53.686165415,[INF]: dimm: 0 2023-08-11T15:02:53.686166988,[INF]: mrank: 1 2023-08-11T15:02:53.686168553,[INF]: srank: 0 2023-08-11T15:02:53.686170143,[INF]: BG: 1 2023-08-11T15:02:53.686171732,[INF]: BA: 0 2023-08-11T15:02:53.686173341,[INF]: row: 0x18731 2023-08-11T15:02:53.686174945,[INF]: col: 0x2a 2023-08-11T15:02:53.686176597,[INF]: beat pair: 6/7 2023-08-11T15:02:53.686178201,[INF]: error mode: 2 2023-08-11T15:02:53.686180095,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.686181660,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.686183230,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.686184913,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.686186618,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.686188361,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.686190063,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.686192527,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 57 2023-08-11T15:02:53.686194157,[INF]: subtest: 3 2023-08-11T15:02:53.686195722,[INF]: subcommand: 0 2023-08-11T15:02:53.686197317,[INF]: dimm: 0 2023-08-11T15:02:53.686198879,[INF]: mrank: 1 2023-08-11T15:02:53.686200452,[INF]: srank: 0 2023-08-11T15:02:53.686202038,[INF]: BG: 2 2023-08-11T15:02:53.686203604,[INF]: BA: 2 2023-08-11T15:02:53.686205234,[INF]: row: 0x0f65b 2023-08-11T15:02:53.686208193,[INF]: col: 0x20 2023-08-11T15:02:53.686209890,[INF]: beat pair: 6/7 2023-08-11T15:02:53.686211476,[INF]: error mode: 2 2023-08-11T15:02:53.686213394,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.686214953,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.686216558,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.686218265,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.686219998,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.686221697,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.686223422,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.686225887,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 58 2023-08-11T15:02:53.686227508,[INF]: subtest: 3 2023-08-11T15:02:53.686229104,[INF]: subcommand: 0 2023-08-11T15:02:53.686230679,[INF]: dimm: 0 2023-08-11T15:02:53.686232244,[INF]: mrank: 1 2023-08-11T15:02:53.686233824,[INF]: srank: 0 2023-08-11T15:02:53.686235422,[INF]: BG: 2 2023-08-11T15:02:53.686237000,[INF]: BA: 0 2023-08-11T15:02:53.686238605,[INF]: row: 0x1e311 2023-08-11T15:02:53.686240215,[INF]: col: 0x25 2023-08-11T15:02:53.686241832,[INF]: beat pair: 6/7 2023-08-11T15:02:53.686243409,[INF]: error mode: 2 2023-08-11T15:02:53.686245293,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.686246870,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.686248478,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.686250147,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.686251857,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.686253547,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.686255252,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.686257709,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 59 2023-08-11T15:02:53.686259347,[INF]: subtest: 3 2023-08-11T15:02:53.686260924,[INF]: subcommand: 0 2023-08-11T15:02:53.686262518,[INF]: dimm: 0 2023-08-11T15:02:53.686264080,[INF]: mrank: 1 2023-08-11T15:02:53.686265660,[INF]: srank: 0 2023-08-11T15:02:53.686267263,[INF]: BG: 0 2023-08-11T15:02:53.686268837,[INF]: BA: 2 2023-08-11T15:02:53.686270437,[INF]: row: 0x0d35b 2023-08-11T15:02:53.686272033,[INF]: col: 0x2e 2023-08-11T15:02:53.686273653,[INF]: beat pair: 6/7 2023-08-11T15:02:53.686275245,[INF]: error mode: 2 2023-08-11T15:02:53.686277139,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.686278712,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.686280311,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.686282001,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.686283722,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.686285438,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.686287143,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.686289577,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 60 2023-08-11T15:02:53.686291201,[INF]: subtest: 3 2023-08-11T15:02:53.686292803,[INF]: subcommand: 0 2023-08-11T15:02:53.686294369,[INF]: dimm: 0 2023-08-11T15:02:53.686295946,[INF]: mrank: 1 2023-08-11T15:02:53.686297526,[INF]: srank: 0 2023-08-11T15:02:53.686299093,[INF]: BG: 3 2023-08-11T15:02:53.686300661,[INF]: BA: 2 2023-08-11T15:02:53.686302266,[INF]: row: 0x0b8c4 2023-08-11T15:02:53.686303872,[INF]: col: 0x3a 2023-08-11T15:02:53.686305514,[INF]: beat pair: 6/7 2023-08-11T15:02:53.686307108,[INF]: error mode: 2 2023-08-11T15:02:53.686308997,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.686310576,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.686312169,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.686313861,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.686315972,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.686317693,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.686319392,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.686321864,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 61 2023-08-11T15:02:53.686323471,[INF]: subtest: 3 2023-08-11T15:02:53.686325053,[INF]: subcommand: 0 2023-08-11T15:02:53.686326649,[INF]: dimm: 0 2023-08-11T15:02:53.686328204,[INF]: mrank: 1 2023-08-11T15:02:53.686329774,[INF]: srank: 0 2023-08-11T15:02:53.686331354,[INF]: BG: 3 2023-08-11T15:02:53.686332934,[INF]: BA: 3 2023-08-11T15:02:53.686334518,[INF]: row: 0x14f13 2023-08-11T15:02:53.686336104,[INF]: col: 0x1f 2023-08-11T15:02:53.686337717,[INF]: beat pair: 6/7 2023-08-11T15:02:53.686339301,[INF]: error mode: 2 2023-08-11T15:02:53.686341188,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.686342767,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.686344338,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.686346044,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.686347742,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.686349439,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.686351141,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.686353611,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 62 2023-08-11T15:02:53.686355236,[INF]: subtest: 3 2023-08-11T15:02:53.686356864,[INF]: subcommand: 0 2023-08-11T15:02:53.686436571,[INF]: dimm: 0 2023-08-11T15:02:53.686438563,[INF]: mrank: 1 2023-08-11T15:02:53.686440161,[INF]: srank: 0 2023-08-11T15:02:53.686441753,[INF]: BG: 1 2023-08-11T15:02:53.686443357,[INF]: BA: 0 2023-08-11T15:02:53.686444975,[INF]: row: 0x1269a 2023-08-11T15:02:53.686446667,[INF]: col: 0x04 2023-08-11T15:02:53.686448297,[INF]: beat pair: 6/7 2023-08-11T15:02:53.686449899,[INF]: error mode: 2 2023-08-11T15:02:53.686452132,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.686453734,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.686455316,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.686457019,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.686458719,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.686460443,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.686462149,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.686464944,[INF]:explorer.mp:k0:n0:s0:p04:c0 error trap: 63 2023-08-11T15:02:53.686466627,[INF]: subtest: 3 2023-08-11T15:02:53.686468231,[INF]: subcommand: 0 2023-08-11T15:02:53.686469819,[INF]: dimm: 0 2023-08-11T15:02:53.686471383,[INF]: mrank: 1 2023-08-11T15:02:53.686473001,[INF]: srank: 0 2023-08-11T15:02:53.686474574,[INF]: BG: 3 2023-08-11T15:02:53.686476148,[INF]: BA: 1 2023-08-11T15:02:53.686477752,[INF]: row: 0x1ba4c 2023-08-11T15:02:53.686479351,[INF]: col: 0x28 2023-08-11T15:02:53.686480970,[INF]: beat pair: 6/7 2023-08-11T15:02:53.686482540,[INF]: error mode: 2 2023-08-11T15:02:53.686484436,[INF]: BYTE : 0011223344556677 8899 2023-08-11T15:02:53.686486002,[INF]: NIBBLE : 0101010101010101 0101 2023-08-11T15:02:53.686487577,[INF]: ERRORS : X............... ..MM 2023-08-11T15:02:53.686489279,[INF]: burst 6 actual : 0xaaaaaaaaaaaaaaaa aa77 2023-08-11T15:02:53.686490984,[INF]: burst 6 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.686492687,[INF]: burst 7 actual : 0x5aaaaaaaaaaaaaaa aaff 2023-08-11T15:02:53.686494388,[INF]: burst 7 expected: 0xaaaaaaaaaaaaaaaa aaaa 2023-08-11T15:02:53.755528975,[TRC]:End MCBIST

dcrowell77 commented 11 months ago

@avalluriMu - fyi

Grubby0624 commented 11 months ago

Hello, thank you for your support. After modifying the values of the following registers, the issue of MCE error has been resolved: https://github.com/open-power/hostboot/blob/fa90493edad80a0b34737cd83267712a88899ce1/src/import/chips/ocmb/explorer/common/include/explorer_scom_addresses.H#L1178 But I still have two questions and I hope you can help me answer them:

  1. The interpretation of this register is "Read to Read to different master". What does the 'master; here mean?
  2. I see that the register addresses related to OCMB are defined in these two files: explorer scom addresses.H explorer scom addresses_fixes.H, what does' fixes' mean here?Are there any differences in the SCOM addresses between these two files? I think after understanding these two issues, we can close this issue. Thank you very much!
ecorderoibm commented 11 months ago
  1. Master in this case is master rank, ie same chip select
  2. those registers have no real difference. the first file is the main one and the _fixes one are ones that were missing from the first one.
Grubby0624 commented 11 months ago

Got it. Thanks again!