System freeze on i7-12700H (06-9a-03, microcode 0x41e and 0x429)

Technologicat commented 1 year ago

Hi,

I have a CLEVO PD70PNN1 laptop with an Alder Lake i7-12700H CPU (06-9a-03). I'm running Linux Mint 21.

The system randomly freezes under certain workloads. When this occurs, nothing responds (not even Alt+SysRq+REISUB), audio starts looping, the last picture remains on the screen, the system instantly vanishes from the local network (no response to ping or ssh), and most often, the system automatically reboots itself after ~10 seconds. Sometimes it doesn't reboot, and I have to hold down the power button to forcibly turn the laptop off.

After the next bootup, the kernel log reports the error 8f87f311-c998-4d9e-a0c4-6065518c4f6d from the previous boot. There is nothing else relevant to the freeze in any of the system logs.

Googling the error number led me here. I read through #44 and #58, but the conditions triggering the freeze as well as the CPU model are different.

The full error message, complete with hex dump, is included at the end of this post.

Any thoughts, anyone? If the error looks relevant to the Intel microcode, what other info should I provide? And in case I'm barking up the wrong tree, any pointers where to go next?

Some details:

The machine is new and has been tested (in Windows) by the retailer I bought it from, so a hardware failure is unlikely. Linux Mint 21 is the only OS on the machine.

In the BIOS, I have set the GPU mode to only use the NVIDIA dGPU (RTX 3070 Ti Mobile). This solved some issues with external displays. The Intel iGPU is off, and lsmod confirms the i915 driver is not loaded.

The (Samsung) SSD firmware is up to date, and setting nvme_core.default_ps_max_latency_us=0 does not affect the freezing.

Workloads that reliably reproduce the system freeze:

Playing Cyberpunk 2077. Steam with GE-Proton. The game otherwise works perfectly (with RTX raytracing enabled, even), and no system freezes have been reported for this game on ProtonDB. Running the game for ~2 minutes usually freezes the system.
Browsing through installed checkpoints in the Automatic1111 GUI for the Stable Diffusion AI art generator. Native Linux app, consisting of a JavaScript frontend, and a Python + Pytorch backend, both running on this machine. Browsing through a couple dozen models usually freezes the system.

In the Automatic1111 workload, when the freeze occurs, GPU compute usage itself (according to nvtop) remains at 0%. The system freezes while Automatic1111 is computing a checksum for the model checkpoint file, on the CPU.

Over the two months the system has been in use, there have also been freezes on rare occasions with other workloads, for example:

Browsing through a directory of pictures in Xviewer, the default picture viewing app in Linux Mint. Happened a few times.
Drawing a simple technical diagram in Inkscape. Happened once.
MS Teams video call, using the new PWA client based on Chrome. Happened once.
Inference (i.e. generating art) in Automatic1111. Happened once.

But curiously, I can go a full work week writing a book in Emacs, recompiling the PDF every few minutes (org-mode + pdflatex), without experiencing any system freezes.

I have gone through the usual troubleshooting checklist: temperatures, RAM health, NVIDIA driver version, SSD firmware version, NVMe power saving, CPU C-states, ACPI, pcie_aspm=off, other miscellaneous kernel boot options, and so on - nothing so far has affected the freezing behavior. I can post the details if needed.

The system behaves the same on its built-in microcode revision 0x41e, as well as with the updated microcode from the 20230214 release (06-9a-03, revision 0x429, date 2023-01-11). The new microcode was installed via apt, package intel-microcode, version 3.20230214.0ubuntu0.22.04.1. Specifically this is from jammy-updates, which is one of the default repositories of Linux Mint 21.

The BIOS is the original version that came preinstalled; CLEVO's driver download page for PD70PNN1 does not have any BIOS updates.

Full error message from the kernel log

[  +0,001533] BERT: Error records from previous boot:
[  +0,001312] [Hardware Error]: event severity: fatal
[  +0,001318] [Hardware Error]:  Error 0, type: fatal
[  +0,001328] [Hardware Error]:   section_type: Firmware Error Record Reference
[  +0,001341] [Hardware Error]:   Firmware Error Record Type: SOC Firmware Error Record Type2
[  +0,001347] [Hardware Error]:   Revision: 2
[  +0,001340] [Hardware Error]:   Record Identifier: 8f87f311-c998-4d9e-a0c4-6065518c4f6d
[  +0,001359] [Hardware Error]:   00000000: 11036101 00000080 00000000 fe013d3c  .a..........<=..
[  +0,001363] [Hardware Error]:   00000010: 00000000 bc570ae8 00002304 bc570e2b  ......W..#..+.W.
[  +0,001361] [Hardware Error]:   00000020: 00002302 bc570f1c 00002303 bc5710b8  .#....W..#....W.
[  +0,001349] [Hardware Error]:   00000030: 00002305 bc571a54 00002300 bd76a86a  .#..T.W..#..j.v.
[  +0,001341] [Hardware Error]:   00000040: 00002301 bd76b1bc 00002304 bd76b4ff  .#....v..#....v.
[  +0,001335] [Hardware Error]:   00000050: 00002302 bd76b5f0 00002303 bd76b788  .#....v..#....v.
[  +0,001322] [Hardware Error]:   00000060: 00002305 bd76c2cc 00002300 bdd83865  .#....v..#..e8..
[  +0,001296] [Hardware Error]:   00000070: 00002301 bdd8439f 00002304 bdd846e2  .#...C...#...F..
[  +0,001275] [Hardware Error]:   00000080: 00002302 bdd847d3 00002303 bdd8496f  .#...G...#..oI..
[  +0,001267] [Hardware Error]:   00000090: 00002305 bdd85300 00002300 becce23d  .#...S...#..=...
[  +0,001254] [Hardware Error]:   000000a0: 00002301 becced77 00002304 beccf0ba  .#..w....#......
[  +0,001242] [Hardware Error]:   000000b0: 00002302 beccf1ab 00002303 beccf347  .#.......#..G...
[  +0,001232] [Hardware Error]:   000000c0: 00002305 beccfcd8 00002300 c005277b  .#.......#..{'..
[  +0,001229] [Hardware Error]:   000000d0: 00002301 c00530d0 00002304 c0053413  .#...0...#...4..
[  +0,001224] [Hardware Error]:   000000e0: 00002302 c0053504 00002303 c00536a0  .#...5...#...6..
[  +0,001216] [Hardware Error]:   000000f0: 00002305 c00541e4 00002300 c05a2423  .#...A...#..#$Z.
[  +0,001209] [Hardware Error]:   00000100: 00002301 c05a2f5d 00002304 c05a9787  .#..]/Z..#....Z.
[  +0,001202] [Hardware Error]:   00000110: 00002302 c05a9878 00002303 c05a9a14  .#..x.Z..#....Z.
[  +0,001194] [Hardware Error]:   00000120: 00002305 c05aa3b0 00000002 01002000  .#....Z...... ..
[  +0,001179] [Hardware Error]:   00000130: c0d236b5 00002014 c0d23801 00002301  .6... ...8...#..
[  +0,001164] [Hardware Error]:   00000140: b838b0eb 00002304 b839887a 00002302  ..8..#..z.9..#..
[  +0,001160] [Hardware Error]:   00000150: b839896b 00002303 b8398b07 00002305  k.9..#....9..#..
[  +0,001134] [Hardware Error]:   00000160: b83994a3 00002300 b9ee3782 00002301  ..9..#...7...#..
[  +0,001115] [Hardware Error]:   00000170: b9ee42bc 00002304 b9ee45ff 00002302  .B...#...E...#..
[  +0,001098] [Hardware Error]:   00000180: b9ee46f0 00002303 b9ee488c 00002305  .F...#...H...#..
[  +0,001070] [Hardware Error]:   00000190: b9ee521d 00002300 babb26aa 00002301  .R...#...&...#..
[  +0,001052] [Hardware Error]:   000001a0: babb2fff 00002304 babb3342 00002302  ./...#..B3...#..
[  +0,001027] [Hardware Error]:   000001b0: babb3433 00002303 babb35cf 00002305  34...#...5...#..
[  +0,001021] [Hardware Error]:   000001c0: babb4271 00002300 bb2c5589 00002301  qB...#...U,..#..
[  +0,001011] [Hardware Error]:   000001d0: bb2c60c3 00002304 bb2c6406 00002302  .`,..#...d,..#..
[  +0,001002] [Hardware Error]:   000001e0: bb2c64f7 00002303 bb2c6693 00002305  .d,..#...f,..#..
[  +0,000991] [Hardware Error]:   000001f0: bb2c7024 00002300 bc56ffae 00002301  $p,..#....V..#..

Technologicat commented 1 year ago

More details posted to the Linux Mint forums, in case the issue could be caused by something else.

Help with decoding the BERT hexdump is still welcome, if anyone knows how to interpret it - it's the only clue. I've collected a few more, in case a larger sample helps.

whpenner commented 1 year ago

@Technologicat Is there more BERT data than you are showing above? The data appears to be missing some data records, so it doesn't have any useful information. If this is all of the data, then there may be a BIOS bug or some other reason that resulted in not capturing all of the BERT data. There is not enough info here to know what may have happened.

Technologicat commented 1 year ago

@whpenner Thanks for taking a look!

In that particular instance of the error, that was all of the data. However, in more recent instances of the error, for some reason, it has produced longer records - long enough for bert.c to skip them, so that I've had to grab them from /sys/firmware/acpi/tables/data/BERT.

Below is a representative collection (7 records, 3300 bytes each). Here's a hexdump -C of each; please tell me if you prefer some other format, or some other way of posting them.

EDIT: Here's a zip file with the raw data, numbered as BERT1 ... BERT7, corresponding to the numbering below.

Instance 1

00000000  00 00 00 00 00 00 00 00  00 00 00 00 d0 0c 00 00  |................|
00000010  01 00 00 00 96 2a 21 81  ed 09 96 49 94 71 8d 72  |.....*!....I.q.r|
00000020  9c 8e 69 ed 01 00 00 00  00 03 00 00 20 0a 00 00  |..i......... ...|
00000030  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000050  00 00 00 00 00 00 00 00  00 00 00 00 02 02 00 00  |................|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 11 f3 87 8f  |................|
00000070  98 c9 9e 4d a0 c4 60 65  51 8c 4f 6d 02 63 03 01  |...M..`eQ.Om.c..|
00000080  80 02 00 00 d0 3b 8a a6  02 00 00 00 00 00 1b 00  |.....;..........|
00000090  05 00 00 00 ff 03 00 80  4e cd 02 00 64 f6 01 00  |........N...d...|
000000a0  24 74 02 00 00 00 00 00  00 00 02 00 00 c7 04 ff  |$t..............|
000000b0  6f ff 8f f0 09 ff 73 60  0a f1 20 60 9f 00 00 11  |o.....s`.. `....|
000000c0  20 00 20 00 00 40 00 01  00 00 00 00 04 00 00 00  | . ..@..........|
000000d0  81 28 50 20 12 cf 0e 00  5d 03 00 01 00 00 00 00  |.(P ....].......|
000000e0  70 00 00 00 00 00 10 01  00 00 00 05 8f 30 70 5a  |p............0pZ|
000000f0  3c 00 20 00 0e 00 00 fc  01 80 c8 39 70 fe fc c7  |<. ........9p...|
00000100  3f c6 4f 7e 47 ff ff 81  db 38 70 7c 00 32 00 0f  |?.O~G....8p|.2..|
00000110  78 41 03 1e 01 c2 30 70  5a 3c 00 00 00 06 00 00  |xA....0pZ<......|
00000120  00 00 00 c0 20 00 00 00  00 00 00 00 00 00 00 00  |.... ...........|
00000130  00 00 20 20 c6 fc 00 20  00 08 00 00 00 00 00 00  |..  ... ........|
00000140  c7 83 00 03 00 c0 39 f0  00 00 00 00 40 22 18 80  |......9.....@"..|
00000150  58 00 80 06 00 00 c0 00  00 00 00 80 10 00 00 00  |X...............|
00000160  40 03 00 b8 14 01 02 00  00 c0 1f aa 8a 88 80 00  |@...............|
00000170  0a 10 01 04 00 1b a3 a0  80 20 00 02 03 fe 05 7f  |......... ......|
00000180  01 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000190  00 00 00 00 00 00 00 62  00 00 00 00 00 00 00 00  |.......b........|
000001a0  00 00 04 00 04 00 00 20  41 33 20 00 90 00 40 00  |....... A3 ...@.|
000001b0  00 00 00 00 80 00 00 01  d0 ff 7f ff ff c8 77 fc  |..............w.|
000001c0  fd fd ff ff 00 00 00 09  e4 ad 90 02 00 00 00 01  |................|
000001d0  ff 0f 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001e0  00 00 72 01 10 34 00 00  00 00 00 00 00 00 00 00  |..r..4..........|
000001f0  00 00 1f 00 00 00 00 00  00 00 00 01 ff 0f 00 00  |................|
00000200  00 00 00 00 00 00 00 00  00 00 00 00 00 00 72 01  |..............r.|
00000210  00 04 00 00 00 00 00 00  00 00 00 00 00 00 1f 00  |................|
00000220  00 00 00 00 00 00 00 01  ff 0f 00 00 00 00 00 00  |................|
00000230  00 00 00 00 00 00 00 00  00 00 72 01 00 04 00 00  |..........r.....|
00000240  00 00 00 00 00 00 00 00  00 00 1f 00 00 00 00 00  |................|
00000250  00 00 00 01 ff 0f 00 00  00 00 00 00 00 00 00 00  |................|
00000260  00 00 00 00 00 00 72 01  00 04 00 00 00 00 00 00  |......r.........|
00000270  00 00 00 00 00 00 1f 00  00 00 00 00 00 00 00 01  |................|
00000280  ff 0f 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000290  00 00 72 01 10 34 00 00  00 00 00 00 00 00 00 00  |..r..4..........|
000002a0  00 00 1f 00 00 00 00 00  e1 1c 18 3d e0 0c cb 0c  |...........=....|
000002b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 72 01  |..............r.|
000002c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 1f 00  |................|
000002d0  00 00 00 00 00 00 00 01  ff 0f 00 00 00 00 00 00  |................|
000002e0  00 00 00 00 00 00 00 00  00 00 72 01 00 04 00 00  |..........r.....|
000002f0  00 00 00 00 00 00 00 00  00 00 1f 00 00 00 00 00  |................|
00000300  86 10 18 3b 85 00 ee 07  00 00 00 00 00 00 00 00  |...;............|
00000310  00 00 00 00 00 00 72 01  00 00 00 00 00 00 00 00  |......r.........|
00000320  00 00 00 00 00 00 1f 00  00 00 00 00 00 00 00 01  |................|
00000330  ff 0f 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000340  00 00 72 01 10 34 00 00  00 00 00 00 00 00 00 00  |..r..4..........|
00000350  00 00 1f 00 00 00 00 00  00 00 00 01 ff 0f 00 00  |................|
00000360  00 00 00 00 00 00 00 00  00 00 00 00 00 00 72 01  |..............r.|
00000370  00 04 00 00 00 00 00 00  00 00 00 00 00 00 1f 00  |................|
00000380  00 00 00 00 00 00 00 01  ff 0f 00 00 00 00 00 00  |................|
00000390  00 00 00 00 00 00 00 00  00 00 72 01 00 04 00 00  |..........r.....|
000003a0  00 00 00 00 00 00 00 00  00 00 1f 00 00 00 00 00  |................|
000003b0  00 00 00 01 ff 0f 00 00  00 00 00 00 00 00 00 00  |................|
000003c0  00 00 00 00 00 00 72 01  00 04 00 00 00 00 00 00  |......r.........|
000003d0  00 00 00 00 00 00 1f 00  00 00 00 00 00 00 00 00  |................|
000003e0  00 00 00 00 00 00 00 00  00 00 20 2f 00 01 00 00  |.......... /....|
000003f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 02  |..............U.|
00000400  00 90 06 05 f1 00 20 00  00 00 00 40 00 00 03 1f  |...... ....@....|
00000410  24 02 cb 03 40 04 00 00  00 00 00 00 00 00 ad de  |$...@...........|
00000420  ef be ad de ef be ad de  ef be ad de ef be ad de  |................|
*
00000a70  ef be ad de ef be ad de  ef be ad de 96 2a 21 81  |.............*!.|
00000a80  ed 09 96 49 94 71 8d 72  9c 8e 69 ed 01 00 00 00  |...I.q.r..i.....|
00000a90  00 03 00 00 20 02 00 00  00 00 00 00 00 00 00 00  |.... ...........|
00000aa0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000ac0  00 00 00 00 02 02 00 00  00 00 00 00 00 00 00 00  |................|
00000ad0  00 00 00 00 11 f3 87 8f  98 c9 9e 4d a0 c4 60 65  |...........M..`e|
00000ae0  51 8c 4f 6d 01 61 03 11  80 00 00 00 00 00 00 00  |Q.Om.a..........|
00000af0  b0 3d 01 fe 00 00 00 00  05 23 00 00 96 0f 57 12  |.=.......#....W.|
00000b00  00 23 00 00 cd ad 56 13  01 23 00 00 07 b9 56 13  |.#....V..#....V.|
00000b10  04 23 00 00 4a bc 56 13  02 23 00 00 3b bd 56 13  |.#..J.V..#..;.V.|
00000b20  03 23 00 00 d7 be 56 13  05 23 00 00 5d c8 56 13  |.#....V..#..].V.|
00000b30  00 23 00 00 7d 99 9e 13  01 23 00 00 cf a2 9e 13  |.#..}....#......|
00000b40  04 23 00 00 12 a6 9e 13  02 23 00 00 03 a7 9e 13  |.#.......#......|
00000b50  03 23 00 00 9b a8 9e 13  05 23 00 00 df b3 9e 13  |.#.......#......|
00000b60  00 23 00 00 df 48 c9 14  01 23 00 00 19 54 c9 14  |.#...H...#...T..|
00000b70  04 23 00 00 00 61 c9 14  02 23 00 00 f1 61 c9 14  |.#...a...#...a..|
00000b80  03 23 00 00 8d 63 c9 14  05 23 00 00 1e 6d c9 14  |.#...c...#...m..|
00000b90  00 23 00 00 58 07 e8 15  01 23 00 00 92 12 e8 15  |.#..X....#......|
00000ba0  04 23 00 00 d5 15 e8 15  02 23 00 00 c6 16 e8 15  |.#.......#......|
00000bb0  03 23 00 00 62 18 e8 15  05 23 00 00 9a 25 e8 15  |.#..b....#...%..|
00000bc0  00 23 00 00 86 64 2b 16  01 23 00 00 db 6d 2b 16  |.#...d+..#...m+.|
00000bd0  04 23 00 00 1e 71 2b 16  02 23 00 00 0f 72 2b 16  |.#...q+..#...r+.|
00000be0  03 23 00 00 ab 73 2b 16  05 23 00 00 ef 7e 2b 16  |.#...s+..#...~+.|
00000bf0  00 23 00 00 c3 48 81 17  01 23 00 00 fd 53 81 17  |.#...H...#...S..|
00000c00  04 23 00 00 40 57 81 17  02 23 00 00 31 58 81 17  |.#..@W...#..1X..|
00000c10  03 23 00 00 cd 59 81 17  05 23 00 00 5e 63 81 17  |.#...Y...#..^c..|
00000c20  00 23 00 00 de c4 73 18  01 23 00 00 18 d0 73 18  |.#....s..#....s.|
00000c30  04 23 00 00 5b d3 73 18  02 23 00 00 4c d4 73 18  |.#..[.s..#..L.s.|
00000c40  03 23 00 00 e8 d5 73 18  05 23 00 00 79 df 73 18  |.#....s..#..y.s.|
00000c50  00 23 00 00 10 e7 f3 18  01 23 00 00 62 f0 f3 18  |.#.......#..b...|
00000c60  04 23 00 00 a5 f3 f3 18  02 23 00 00 96 f4 f3 18  |.#.......#......|
00000c70  03 23 00 00 2e f6 f3 18  05 23 00 00 72 01 f4 18  |.#.......#..r...|
00000c80  02 00 00 00 00 20 00 01  9f a9 39 19 14 20 00 00  |..... ....9.. ..|
00000c90  eb aa 39 19 01 23 00 00  a5 73 50 11 04 23 00 00  |..9..#...sP..#..|
00000ca0  e8 76 50 11 02 23 00 00  d9 77 50 11 03 23 00 00  |.vP..#...wP..#..|
00000cb0  75 79 50 11 05 23 00 00  b9 84 50 11 00 23 00 00  |uyP..#....P..#..|
00000cc0  06 f5 56 12 01 23 00 00  40 00 57 12 04 23 00 00  |..V..#..@.W..#..|
00000cd0  83 03 57 12 02 23 00 00  74 04 57 12 03 23 00 00  |..W..#..t.W..#..|
00000ce0  10 06 57 12                                       |..W.|
00000ce4

Instance 2

00000000  00 00 00 00 00 00 00 00  00 00 00 00 d0 0c 00 00  |................|
00000010  01 00 00 00 96 2a 21 81  ed 09 96 49 94 71 8d 72  |.....*!....I.q.r|
00000020  9c 8e 69 ed 01 00 00 00  00 03 00 00 20 0a 00 00  |..i......... ...|
00000030  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000050  00 00 00 00 00 00 00 00  00 00 00 00 02 02 00 00  |................|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 11 f3 87 8f  |................|
00000070  98 c9 9e 4d a0 c4 60 65  51 8c 4f 6d 02 63 03 01  |...M..`eQ.Om.c..|
00000080  80 02 00 00 28 6e 90 16  03 00 00 00 00 00 1b 00  |....(n..........|
00000090  05 00 00 00 ff 03 00 80  68 f6 01 00 44 70 03 00  |........h...Dp..|
000000a0  24 74 02 00 00 00 00 00  00 00 02 00 00 c7 04 ff  |$t..............|
000000b0  6f ff 8f f0 09 ff 63 60  0a f1 20 60 9f 00 00 11  |o.....c`.. `....|
000000c0  20 00 20 00 00 40 00 01  00 00 00 00 04 00 00 00  | . ..@..........|
000000d0  81 28 50 20 12 cf 0e 00  5d 03 04 03 41 00 00 00  |.(P ....]...A...|
000000e0  70 00 00 00 00 00 10 01  00 00 00 05 8f 30 70 5a  |p............0pZ|
000000f0  3c 00 20 00 0e 00 00 fc  01 80 c8 39 70 fe fc c7  |<. ........9p...|
00000100  3f c6 4f 7e 47 ff ff 81  db 38 70 7c 00 32 00 0f  |?.O~G....8p|.2..|
00000110  78 41 03 1e 01 c2 30 70  5a 3c 00 00 00 06 00 00  |xA....0pZ<......|
00000120  00 00 00 c0 20 00 00 00  00 00 00 00 00 00 00 00  |.... ...........|
00000130  00 00 20 20 c6 fc 00 20  00 08 00 00 00 00 00 00  |..  ... ........|
00000140  c7 83 00 03 00 c0 39 f0  00 00 00 00 40 22 18 80  |......9.....@"..|
00000150  58 00 80 06 00 00 c0 00  00 00 00 80 10 00 00 00  |X...............|
00000160  40 03 00 b8 14 01 02 00  00 c0 1f aa 8a 88 80 00  |@...............|
00000170  0a 10 01 04 00 1b a3 a0  80 20 00 02 03 fe 05 7f  |......... ......|
00000180  01 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000190  00 00 00 00 00 00 00 62  00 00 00 00 00 00 00 00  |.......b........|
000001a0  00 00 04 00 04 00 00 20  41 33 20 00 90 00 40 00  |....... A3 ...@.|
000001b0  00 00 00 00 80 00 00 01  d0 ff 7f ff ff c8 77 fc  |..............w.|
000001c0  fd fd ff ff 00 00 00 09  e4 ad 90 02 00 00 00 01  |................|
000001d0  ff 0f 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001e0  00 00 72 01 10 34 00 00  00 00 00 00 00 00 00 00  |..r..4..........|
000001f0  00 00 1f 00 00 00 00 00  00 00 00 01 ff 0f 00 00  |................|
00000200  00 00 00 00 00 00 00 00  00 00 00 00 00 00 72 01  |..............r.|
00000210  00 04 00 00 00 00 00 00  00 00 00 00 00 00 1f 00  |................|
00000220  00 00 00 00 00 00 00 01  ff 0f 00 00 00 00 00 00  |................|
00000230  00 00 00 00 00 00 00 00  00 00 72 01 00 04 00 00  |..........r.....|
00000240  00 00 00 00 00 00 00 00  00 00 1f 00 00 00 00 00  |................|
00000250  00 00 00 01 ff 0f 00 00  00 00 00 00 00 00 00 00  |................|
00000260  00 00 00 00 00 00 72 01  00 04 00 00 00 00 00 00  |......r.........|
00000270  00 00 00 00 00 00 1f 00  00 00 00 00 00 00 00 01  |................|
00000280  ff 0f 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000290  00 00 72 01 10 34 00 00  00 00 00 00 00 00 00 00  |..r..4..........|
000002a0  00 00 1f 00 00 00 00 00  df 1c 18 3d de 0c c9 0c  |...........=....|
000002b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 72 01  |..............r.|
000002c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 1f 00  |................|
000002d0  00 00 00 00 00 00 00 01  ff 0f 00 00 00 00 00 00  |................|
000002e0  00 00 00 00 00 00 00 00  00 00 72 01 00 04 00 00  |..........r.....|
000002f0  00 00 00 00 00 00 00 00  00 00 1f 00 00 00 00 00  |................|
00000300  30 11 18 3b 2f 01 d3 09  00 00 00 00 00 00 00 00  |0..;/...........|
00000310  00 00 00 00 00 00 72 01  00 00 00 00 00 00 00 00  |......r.........|
00000320  00 00 00 00 00 00 1f 00  00 00 00 00 00 00 00 01  |................|
00000330  ff 0f 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000340  00 00 72 01 10 34 00 00  00 00 00 00 00 00 00 00  |..r..4..........|
00000350  00 00 1f 00 00 00 00 00  00 00 00 01 ff 0f 00 00  |................|
00000360  00 00 00 00 00 00 00 00  00 00 00 00 00 00 72 01  |..............r.|
00000370  00 04 00 00 00 00 00 00  00 00 00 00 00 00 1f 00  |................|
00000380  00 00 00 00 00 00 00 01  ff 0f 00 00 00 00 00 00  |................|
00000390  00 00 00 00 00 00 00 00  00 00 72 01 00 04 00 00  |..........r.....|
000003a0  00 00 00 00 00 00 00 00  00 00 1f 00 00 00 00 00  |................|
000003b0  00 00 00 01 ff 0f 00 00  00 00 00 00 00 00 00 00  |................|
000003c0  00 00 00 00 00 00 72 01  00 04 00 00 00 00 00 00  |......r.........|
000003d0  00 00 00 00 00 00 1f 00  00 00 00 00 00 00 00 00  |................|
000003e0  00 00 00 00 00 00 00 00  00 00 20 2f 00 01 00 00  |.......... /....|
000003f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 02  |..............U.|
00000400  00 90 06 05 f1 0b 20 00  00 00 00 40 00 00 03 1f  |...... ....@....|
00000410  24 02 cb 03 40 04 00 00  00 00 00 00 00 00 ad de  |$...@...........|
00000420  ef be ad de ef be ad de  ef be ad de ef be ad de  |................|
*
00000a70  ef be ad de ef be ad de  ef be ad de 96 2a 21 81  |.............*!.|
00000a80  ed 09 96 49 94 71 8d 72  9c 8e 69 ed 01 00 00 00  |...I.q.r..i.....|
00000a90  00 03 00 00 20 02 00 00  00 00 00 00 00 00 00 00  |.... ...........|
00000aa0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000ac0  00 00 00 00 02 02 00 00  00 00 00 00 00 00 00 00  |................|
00000ad0  00 00 00 00 11 f3 87 8f  98 c9 9e 4d a0 c4 60 65  |...........M..`e|
00000ae0  51 8c 4f 6d 01 61 03 11  80 00 00 00 00 00 00 00  |Q.Om.a..........|
00000af0  d0 3d 01 fe 00 00 00 00  01 23 00 00 f7 87 b6 0d  |.=.......#......|
00000b00  04 23 00 00 3a 8b b6 0d  02 23 00 00 2b 8c b6 0d  |.#..:....#..+...|
00000b10  03 23 00 00 c7 8d b6 0d  05 23 00 00 63 97 b6 0d  |.#.......#..c...|
00000b20  00 23 00 00 d4 06 21 0f  01 23 00 00 0b 12 21 0f  |.#....!..#....!.|
00000b30  04 23 00 00 4e 15 21 0f  02 23 00 00 3f 16 21 0f  |.#..N.!..#..?.!.|
00000b40  03 23 00 00 d7 17 21 0f  05 23 00 00 68 21 21 0f  |.#....!..#..h!!.|
00000b50  00 23 00 00 f5 c6 6f 0f  01 23 00 00 4a d0 6f 0f  |.#....o..#..J.o.|
00000b60  04 23 00 00 8d d3 6f 0f  02 23 00 00 7e d4 6f 0f  |.#....o..#..~.o.|
00000b70  03 23 00 00 1a d6 6f 0f  05 23 00 00 5e e1 6f 0f  |.#....o..#..^.o.|
00000b80  00 23 00 00 4e 89 3f 10  01 23 00 00 88 94 3f 10  |.#..N.?..#....?.|
00000b90  04 23 00 00 cb 97 3f 10  02 23 00 00 bc 98 3f 10  |.#....?..#....?.|
00000ba0  03 23 00 00 58 9a 3f 10  05 23 00 00 e9 a3 3f 10  |.#..X.?..#....?.|
00000bb0  00 23 00 00 29 9c b3 11  01 23 00 00 63 a7 b3 11  |.#..)....#..c...|
00000bc0  04 23 00 00 a6 aa b3 11  02 23 00 00 97 ab b3 11  |.#.......#......|
00000bd0  03 23 00 00 33 ad b3 11  05 23 00 00 b9 b6 b3 11  |.#..3....#......|
00000be0  00 23 00 00 cd dc 09 12  01 23 00 00 22 e6 09 12  |.#.......#.."...|
00000bf0  04 23 00 00 41 59 0a 12  02 23 00 00 32 5a 0a 12  |.#..AY...#..2Z..|
00000c00  03 23 00 00 ce 5b 0a 12  05 23 00 00 12 67 0a 12  |.#...[...#...g..|
00000c10  00 23 00 00 a8 72 da 12  01 23 00 00 e2 7d da 12  |.#...r...#...}..|
00000c20  04 23 00 00 25 81 da 12  02 23 00 00 16 82 da 12  |.#..%....#......|
00000c30  03 23 00 00 b2 83 da 12  05 23 00 00 43 8d da 12  |.#.......#..C...|
00000c40  00 23 00 00 df 8e 74 14  01 23 00 00 16 9a 74 14  |.#....t..#....t.|
00000c50  04 23 00 00 59 9d 74 14  02 23 00 00 4a 9e 74 14  |.#..Y.t..#..J.t.|
00000c60  03 23 00 00 e2 9f 74 14  05 23 00 00 68 a9 74 14  |.#....t..#..h.t.|
00000c70  00 23 00 00 20 3d 9e 14  01 23 00 00 75 46 9e 14  |.#.. =...#..uF..|
00000c80  04 23 00 00 b8 49 9e 14  02 23 00 00 a9 4a 9e 14  |.#...I...#...J..|
00000c90  03 23 00 00 45 4c 9e 14  05 23 00 00 89 57 9e 14  |.#..EL...#...W..|
00000ca0  02 00 00 00 00 20 00 01  15 86 ba 14 14 20 00 00  |..... ....... ..|
00000cb0  46 90 ba 14 01 23 00 00  aa c9 c7 0c 04 23 00 00  |F....#.......#..|
00000cc0  ed cc c7 0c 02 23 00 00  de cd c7 0c 03 23 00 00  |.....#.......#..|
00000cd0  7a cf c7 0c 05 23 00 00  be da c7 0c 00 23 00 00  |z....#.......#..|
00000ce0  bd 7c b6 0d                                       |.|..|
00000ce4

Instance 3

00000000  00 00 00 00 00 00 00 00  00 00 00 00 d0 0c 00 00  |................|
00000010  01 00 00 00 96 2a 21 81  ed 09 96 49 94 71 8d 72  |.....*!....I.q.r|
00000020  9c 8e 69 ed 01 00 00 00  00 03 00 00 20 0a 00 00  |..i......... ...|
00000030  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000050  00 00 00 00 00 00 00 00  00 00 00 00 02 02 00 00  |................|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 11 f3 87 8f  |................|
00000070  98 c9 9e 4d a0 c4 60 65  51 8c 4f 6d 02 63 03 01  |...M..`eQ.Om.c..|
00000080  80 02 00 00 b7 c8 44 91  09 00 00 00 00 00 1b 00  |......D.........|
00000090  05 00 00 00 ff 03 00 80  68 f6 01 00 64 f6 01 00  |........h...d...|
000000a0  24 74 02 00 00 00 00 00  00 00 02 00 00 c7 04 ff  |$t..............|
000000b0  6f ff 8f f0 09 ff 63 60  0a f1 20 60 9f 00 00 11  |o.....c`.. `....|
000000c0  20 00 20 00 00 40 00 01  00 00 00 00 04 00 00 00  | . ..@..........|
000000d0  81 28 50 20 12 cf 0e 00  5d 03 04 03 41 00 00 00  |.(P ....]...A...|
000000e0  70 00 00 00 00 00 10 01  00 00 00 05 8f 30 70 5a  |p............0pZ|
000000f0  3c 00 20 00 0e 00 00 fc  01 80 c8 39 70 fe fc c7  |<. ........9p...|
00000100  3f c6 4f 7e 47 ff ff 81  db 38 70 7c 00 32 00 0f  |?.O~G....8p|.2..|
00000110  78 41 03 1e 01 c2 30 70  5a 3c 00 00 00 06 00 00  |xA....0pZ<......|
00000120  00 00 00 c0 20 00 00 00  00 00 00 00 00 00 00 00  |.... ...........|
00000130  00 00 20 20 c6 fc 00 20  00 08 00 00 00 00 00 00  |..  ... ........|
00000140  c7 83 00 03 00 c0 39 f0  00 00 00 00 40 22 18 80  |......9.....@"..|
00000150  58 00 80 06 00 00 c0 00  00 00 00 80 10 00 00 00  |X...............|
00000160  40 03 00 b8 14 01 02 00  00 c0 1f aa 8a 88 80 00  |@...............|
00000170  0a 10 01 04 00 1b a3 a0  80 20 00 02 03 fe 05 7f  |......... ......|
00000180  01 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000190  00 00 00 00 00 00 00 62  00 00 00 00 00 00 00 00  |.......b........|
000001a0  00 00 04 00 04 00 00 20  41 33 20 00 90 00 40 00  |....... A3 ...@.|
000001b0  00 00 00 00 80 00 00 01  d0 ff 7f ff ff c8 77 fc  |..............w.|
000001c0  fd fd ff ff 00 00 00 09  e4 ad 90 02 00 00 00 01  |................|
000001d0  ff 0f 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001e0  00 00 72 01 10 34 00 00  00 00 00 00 00 00 00 00  |..r..4..........|
000001f0  00 00 1f 00 00 00 00 00  00 00 00 01 ff 0f 00 00  |................|
00000200  00 00 00 00 00 00 00 00  00 00 00 00 00 00 72 01  |..............r.|
00000210  00 04 00 00 00 00 00 00  00 00 00 00 00 00 1f 00  |................|
00000220  00 00 00 00 00 00 00 01  ff 0f 00 00 00 00 00 00  |................|
00000230  00 00 00 00 00 00 00 00  00 00 72 01 00 04 00 00  |..........r.....|
00000240  00 00 00 00 00 00 00 00  00 00 1f 00 00 00 00 00  |................|
00000250  00 00 00 01 ff 0f 00 00  00 00 00 00 00 00 00 00  |................|
00000260  00 00 00 00 00 00 72 01  00 04 00 00 00 00 00 00  |......r.........|
00000270  00 00 00 00 00 00 1f 00  00 00 00 00 00 00 00 01  |................|
00000280  ff 0f 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000290  00 00 72 01 10 34 00 00  00 00 00 00 00 00 00 00  |..r..4..........|
000002a0  00 00 1f 00 00 00 00 00  17 1d 18 3d 16 0d 01 0d  |...........=....|
000002b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 72 01  |..............r.|
000002c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 1f 00  |................|
000002d0  00 00 00 00 00 00 00 01  ff 0f 00 00 00 00 00 00  |................|
000002e0  00 00 00 00 00 00 00 00  00 00 72 01 00 04 00 00  |..........r.....|
000002f0  00 00 00 00 00 00 00 00  00 00 1f 00 00 00 00 00  |................|
00000300  f2 1c 18 3b f1 0c e8 09  00 00 00 00 00 00 00 00  |...;............|
00000310  00 00 00 00 00 00 72 01  00 00 00 00 00 00 00 00  |......r.........|
00000320  00 00 00 00 00 00 1f 00  00 00 00 00 00 00 00 01  |................|
00000330  ff 0f 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000340  00 00 72 01 10 34 00 00  00 00 00 00 00 00 00 00  |..r..4..........|
00000350  00 00 1f 00 00 00 00 00  00 00 00 01 ff 0f 00 00  |................|
00000360  00 00 00 00 00 00 00 00  00 00 00 00 00 00 72 01  |..............r.|
00000370  00 04 00 00 00 00 00 00  00 00 00 00 00 00 1f 00  |................|
00000380  00 00 00 00 00 00 00 01  ff 0f 00 00 00 00 00 00  |................|
00000390  00 00 00 00 00 00 00 00  00 00 72 01 00 04 00 00  |..........r.....|
000003a0  00 00 00 00 00 00 00 00  00 00 1f 00 00 00 00 00  |................|
000003b0  00 00 00 01 ff 0f 00 00  00 00 00 00 00 00 00 00  |................|
000003c0  00 00 00 00 00 00 72 01  00 04 00 00 00 00 00 00  |......r.........|
000003d0  00 00 00 00 00 00 1f 00  00 00 00 00 00 00 00 00  |................|
000003e0  00 00 00 00 00 00 00 00  00 00 20 2f 00 01 00 00  |.......... /....|
000003f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 02  |..............U.|
00000400  00 90 06 05 f1 09 20 00  00 00 00 40 00 00 03 1f  |...... ....@....|
00000410  24 02 cb 03 40 04 00 00  00 00 00 00 00 00 ad de  |$...@...........|
00000420  ef be ad de ef be ad de  ef be ad de ef be ad de  |................|
*
00000a70  ef be ad de ef be ad de  ef be ad de 96 2a 21 81  |.............*!.|
00000a80  ed 09 96 49 94 71 8d 72  9c 8e 69 ed 01 00 00 00  |...I.q.r..i.....|
00000a90  00 03 00 00 20 02 00 00  00 00 00 00 00 00 00 00  |.... ...........|
00000aa0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000ac0  00 00 00 00 02 02 00 00  00 00 00 00 00 00 00 00  |................|
00000ad0  00 00 00 00 11 f3 87 8f  98 c9 9e 4d a0 c4 60 65  |...........M..`e|
00000ae0  51 8c 4f 6d 01 61 03 11  80 00 00 00 00 00 00 00  |Q.Om.a..........|
00000af0  38 3c 01 fe 00 00 00 00  03 23 00 00 e7 44 28 d3  |8<.......#...D(.|
00000b00  05 23 00 00 2b 50 28 d3  02 00 00 00 00 20 00 01  |.#..+P(...... ..|
00000b10  30 c9 2f d3 14 20 00 00  7c ca 2f d3 01 23 00 00  |0./.. ..|./..#..|
00000b20  17 b0 c1 cc 04 23 00 00  5a b3 c1 cc 02 23 00 00  |.....#..Z....#..|
00000b30  4b b4 c1 cc 03 23 00 00  e3 b5 c1 cc 05 23 00 00  |K....#.......#..|
00000b40  27 c1 c1 cc 00 23 00 00  fc 88 d0 cc 01 23 00 00  |'....#.......#..|
00000b50  36 94 d0 cc 04 23 00 00  79 97 d0 cc 02 23 00 00  |6....#..y....#..|
00000b60  6a 98 d0 cc 03 23 00 00  06 9a d0 cc 05 23 00 00  |j....#.......#..|
00000b70  8c a3 d0 cc 00 23 00 00  22 85 d8 cd 01 23 00 00  |.....#.."....#..|
00000b80  5c 90 d8 cd 04 23 00 00  9f 93 d8 cd 02 23 00 00  |\....#.......#..|
00000b90  90 94 d8 cd 03 23 00 00  2c 96 d8 cd 05 23 00 00  |.....#..,....#..|
00000ba0  bd 9f d8 cd 00 23 00 00  d6 fc e4 ce 01 23 00 00  |.....#.......#..|
00000bb0  2b 06 e5 ce 04 23 00 00  5c 66 e6 ce 02 23 00 00  |+....#..\f...#..|
00000bc0  4d 67 e6 ce 03 23 00 00  e9 68 e6 ce 05 23 00 00  |Mg...#...h...#..|
00000bd0  dc 74 e6 ce 00 23 00 00  4b d3 f8 ce 01 23 00 00  |.t...#..K....#..|
00000be0  85 de f8 ce 04 23 00 00  c8 e1 f8 ce 02 23 00 00  |.....#.......#..|
00000bf0  b9 e2 f8 ce 03 23 00 00  55 e4 f8 ce 05 23 00 00  |.....#..U....#..|
00000c00  db ed f8 ce 00 23 00 00  6a 8a 15 d0 01 23 00 00  |.....#..j....#..|
00000c10  a4 95 15 d0 04 23 00 00  e7 98 15 d0 02 23 00 00  |.....#.......#..|
00000c20  d8 99 15 d0 03 23 00 00  74 9b 15 d0 05 23 00 00  |.....#..t....#..|
00000c30  05 a5 15 d0 00 23 00 00  d2 65 09 d1 01 23 00 00  |.....#...e...#..|
00000c40  24 6f 09 d1 04 23 00 00  67 72 09 d1 02 23 00 00  |$o...#..gr...#..|
00000c50  58 73 09 d1 03 23 00 00  f0 74 09 d1 05 23 00 00  |Xs...#...t...#..|
00000c60  34 80 09 d1 00 23 00 00  db 13 20 d1 01 23 00 00  |4....#.... ..#..|
00000c70  15 1f 20 d1 04 23 00 00  58 22 20 d1 02 23 00 00  |.. ..#..X" ..#..|
00000c80  49 23 20 d1 03 23 00 00  e5 24 20 d1 05 23 00 00  |I# ..#...$ ..#..|
00000c90  6b 2e 20 d1 00 23 00 00  48 6d 39 d2 01 23 00 00  |k. ..#..Hm9..#..|
00000ca0  82 78 39 d2 04 23 00 00  c5 7b 39 d2 02 23 00 00  |.x9..#...{9..#..|
00000cb0  b6 7c 39 d2 03 23 00 00  52 7e 39 d2 05 23 00 00  |.|9..#..R~9..#..|
00000cc0  e3 87 39 d2 00 23 00 00  c2 35 28 d3 01 23 00 00  |..9..#...5(..#..|
00000cd0  17 3f 28 d3 04 23 00 00  5a 42 28 d3 02 23 00 00  |.?(..#..ZB(..#..|
00000ce0  4b 43 28 d3                                       |KC(.|
00000ce4

Instance 4

00000000  00 00 00 00 00 00 00 00  00 00 00 00 d0 0c 00 00  |................|
00000010  01 00 00 00 96 2a 21 81  ed 09 96 49 94 71 8d 72  |.....*!....I.q.r|
00000020  9c 8e 69 ed 01 00 00 00  00 03 00 00 20 0a 00 00  |..i......... ...|
00000030  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000050  00 00 00 00 00 00 00 00  00 00 00 00 02 02 00 00  |................|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 11 f3 87 8f  |................|
00000070  98 c9 9e 4d a0 c4 60 65  51 8c 4f 6d 02 63 03 01  |...M..`eQ.Om.c..|
00000080  80 02 00 00 14 aa 5a 49  00 00 00 00 00 00 1b 00  |......ZI........|
00000090  05 00 00 00 ff 03 00 80  68 f6 01 00 64 f6 01 00  |........h...d...|
000000a0  24 74 02 00 00 04 00 00  01 00 02 00 00 c7 04 7f  |$t..............|
000000b0  67 ff 8f f0 09 ff 63 60  0a f1 20 60 9f 00 00 11  |g.....c`.. `....|
000000c0  20 00 20 00 00 40 00 01  00 00 80 00 04 00 00 00  | . ..@..........|
000000d0  81 08 50 60 12 cf 0e 00  5d 03 04 03 41 00 01 00  |..P`....]...A...|
000000e0  70 00 00 00 00 00 10 01  00 00 00 05 8f 30 70 5a  |p............0pZ|
000000f0  3c 00 20 00 0e 00 00 fc  01 80 c8 39 70 fe fc c7  |<. ........9p...|
00000100  3f c6 4f 7e 47 ff ff 81  db 38 70 7c 00 32 00 0f  |?.O~G....8p|.2..|
00000110  78 41 03 1e 01 c2 30 70  5a 3c 00 00 00 06 00 00  |xA....0pZ<......|
00000120  00 00 00 c0 20 00 00 00  00 00 00 00 00 00 00 00  |.... ...........|
00000130  00 00 20 20 c6 fc 00 20  00 08 00 00 00 00 00 00  |..  ... ........|
00000140  c7 83 00 03 00 c0 39 f0  00 00 00 00 40 22 18 80  |......9.....@"..|
00000150  58 00 80 06 00 00 c0 00  00 00 00 80 10 00 00 00  |X...............|
00000160  40 03 00 b8 14 01 02 00  00 c0 1f aa 8a 88 80 00  |@...............|
00000170  0a 10 01 04 00 1b a3 a0  80 20 00 02 03 fe 05 7f  |......... ......|
00000180  01 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000190  00 00 00 00 00 00 00 62  00 00 00 00 00 00 00 00  |.......b........|
000001a0  00 00 04 00 04 00 00 00  01 33 20 00 10 00 00 00  |.........3 .....|
000001b0  00 00 00 00 80 00 00 01  d0 ff 7f ff ff c8 77 fc  |..............w.|
000001c0  fd fd ff ff 00 00 00 09  e4 ad 90 02 00 00 00 01  |................|
000001d0  ff 0f 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001e0  00 00 72 01 10 34 00 00  00 00 00 00 00 00 00 00  |..r..4..........|
000001f0  00 00 1f 00 00 00 00 00  00 00 00 01 ff 0f 00 00  |................|
00000200  00 00 00 00 00 00 00 00  00 00 00 00 00 00 72 01  |..............r.|
00000210  00 04 00 00 00 00 00 00  00 00 00 00 00 00 1f 00  |................|
00000220  00 00 00 00 00 00 00 01  ff 0f 00 00 00 00 00 00  |................|
00000230  00 00 00 00 00 00 00 00  00 00 72 01 00 04 00 00  |..........r.....|
00000240  00 00 00 00 00 00 00 00  00 00 1f 00 00 00 00 00  |................|
00000250  00 00 00 01 ff 0f 00 00  00 00 00 00 00 00 00 00  |................|
00000260  00 00 00 00 00 00 72 01  00 04 00 00 00 00 00 00  |......r.........|
00000270  00 00 00 00 00 00 1f 00  00 00 00 00 00 00 00 01  |................|
00000280  ff 0f 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000290  00 00 72 01 10 34 00 00  00 00 00 00 00 00 00 00  |..r..4..........|
000002a0  00 00 1f 00 00 00 00 00  ad 11 18 3d ac 01 a4 01  |...........=....|
000002b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 72 01  |..............r.|
000002c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 1f 00  |................|
000002d0  00 00 00 00 00 00 00 01  ff 0f 00 00 00 00 00 00  |................|
000002e0  00 00 00 00 00 00 00 00  00 00 72 01 00 04 00 00  |..........r.....|
000002f0  00 00 00 00 00 00 00 00  00 00 1f 00 00 00 00 00  |................|
00000300  50 14 18 3b 4f 04 8a 04  00 00 00 00 00 00 00 00  |P..;O...........|
00000310  00 00 00 00 00 00 72 01  00 00 00 00 00 00 00 00  |......r.........|
00000320  00 00 00 00 00 00 1f 00  00 00 00 00 00 00 00 01  |................|
00000330  ff 0f 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000340  00 00 72 01 10 34 00 00  00 00 00 00 00 00 00 00  |..r..4..........|
00000350  00 00 1f 00 00 00 00 00  00 00 00 01 ff 0f 00 00  |................|
00000360  00 00 00 00 00 00 00 00  00 00 00 00 00 00 72 01  |..............r.|
00000370  00 04 00 00 00 00 00 00  00 00 00 00 00 00 1f 00  |................|
00000380  00 00 00 00 00 00 00 01  ff 0f 00 00 00 00 00 00  |................|
00000390  00 00 00 00 00 00 00 00  00 00 72 01 00 04 00 00  |..........r.....|
000003a0  00 00 00 00 00 00 00 00  00 00 1f 00 00 00 00 00  |................|
000003b0  00 00 00 01 ff 0f 00 00  00 00 00 00 00 00 00 00  |................|
000003c0  00 00 00 00 00 00 72 01  00 04 00 00 00 00 00 00  |......r.........|
000003d0  00 00 00 00 00 00 1f 00  00 00 00 00 00 00 00 00  |................|
000003e0  00 00 00 00 00 00 00 00  00 00 20 2f 00 01 00 00  |.......... /....|
000003f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 02  |..............U.|
00000400  00 90 06 05 f1 00 20 00  00 00 00 44 00 00 03 1f  |...... ....D....|
00000410  24 02 cb 03 40 04 00 00  00 00 00 00 00 00 ad de  |$...@...........|
00000420  ef be ad de ef be ad de  ef be ad de ef be ad de  |................|
*
00000a70  ef be ad de ef be ad de  ef be ad de 96 2a 21 81  |.............*!.|
00000a80  ed 09 96 49 94 71 8d 72  9c 8e 69 ed 01 00 00 00  |...I.q.r..i.....|
00000a90  00 03 00 00 20 02 00 00  00 00 00 00 00 00 00 00  |.... ...........|
00000aa0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000ac0  00 00 00 00 02 02 00 00  00 00 00 00 00 00 00 00  |................|
00000ad0  00 00 00 00 11 f3 87 8f  98 c9 9e 4d a0 c4 60 65  |...........M..`e|
00000ae0  51 8c 4f 6d 01 61 03 11  80 00 00 00 00 00 00 00  |Q.Om.a..........|
00000af0  38 3c 01 fe 00 00 00 00  03 23 00 00 de e3 8e 0a  |8<.......#......|
00000b00  05 23 00 00 64 ed 8e 0a  02 00 00 00 00 20 00 01  |.#..d........ ..|
00000b10  2e a1 8b 0b 14 20 00 00  7a a2 8b 0b 01 23 00 00  |..... ..z....#..|
00000b20  d3 10 95 02 04 23 00 00  15 14 95 02 02 23 00 00  |.....#.......#..|
00000b30  06 15 95 02 03 23 00 00  a2 16 95 02 05 23 00 00  |.....#.......#..|
00000b40  3e 20 95 02 00 23 00 00  eb 06 ac 03 01 23 00 00  |> ...#.......#..|
00000b50  25 12 ac 03 04 23 00 00  68 15 ac 03 02 23 00 00  |%....#..h....#..|
00000b60  59 16 ac 03 03 23 00 00  f5 17 ac 03 05 23 00 00  |Y....#.......#..|
00000b70  86 21 ac 03 00 23 00 00  c8 0f c3 04 01 23 00 00  |.!...#.......#..|
00000b80  1d 19 c3 04 04 23 00 00  60 1c c3 04 02 23 00 00  |.....#..`....#..|
00000b90  51 1d c3 04 03 23 00 00  ed 1e c3 04 05 23 00 00  |Q....#.......#..|
00000ba0  31 2a c3 04 00 23 00 00  7e b0 4d 05 01 23 00 00  |1*...#..~.M..#..|
00000bb0  b5 bb 4d 05 04 23 00 00  f8 be 4d 05 02 23 00 00  |..M..#....M..#..|
00000bc0  e9 bf 4d 05 03 23 00 00  81 c1 4d 05 05 23 00 00  |..M..#....M..#..|
00000bd0  12 cb 4d 05 00 23 00 00  f7 82 2e 06 01 23 00 00  |..M..#.......#..|
00000be0  31 8e 2e 06 04 23 00 00  74 91 2e 06 02 23 00 00  |1....#..t....#..|
00000bf0  65 92 2e 06 03 23 00 00  01 94 2e 06 05 23 00 00  |e....#.......#..|
00000c00  92 9d 2e 06 00 23 00 00  73 e7 55 07 01 23 00 00  |.....#..s.U..#..|
00000c10  c8 f0 55 07 04 23 00 00  0a f4 55 07 02 23 00 00  |..U..#....U..#..|
00000c20  fb f4 55 07 03 23 00 00  97 f6 55 07 05 23 00 00  |..U..#....U..#..|
00000c30  db 01 56 07 00 23 00 00  30 eb e3 07 01 23 00 00  |..V..#..0....#..|
00000c40  6a f6 e3 07 04 23 00 00  ad f9 e3 07 02 23 00 00  |j....#.......#..|
00000c50  9e fa e3 07 03 23 00 00  3a fc e3 07 05 23 00 00  |.....#..:....#..|
00000c60  5c 0e e4 07 00 23 00 00  f1 37 36 09 01 23 00 00  |\....#...76..#..|
00000c70  2b 43 36 09 04 23 00 00  6e 46 36 09 02 23 00 00  |+C6..#..nF6..#..|
00000c80  5f 47 36 09 03 23 00 00  fb 48 36 09 05 23 00 00  |_G6..#...H6..#..|
00000c90  8c 52 36 09 00 23 00 00  2b b6 41 0a 01 23 00 00  |.R6..#..+.A..#..|
00000ca0  80 bf 41 0a 04 23 00 00  c3 c2 41 0a 02 23 00 00  |..A..#....A..#..|
00000cb0  b4 c3 41 0a 03 23 00 00  50 c5 41 0a 05 23 00 00  |..A..#..P.A..#..|
00000cc0  94 d0 41 0a 00 23 00 00  db d2 8e 0a 01 23 00 00  |..A..#.......#..|
00000cd0  12 de 8e 0a 04 23 00 00  55 e1 8e 0a 02 23 00 00  |.....#..U....#..|
00000ce0  46 e2 8e 0a                                       |F...|
00000ce4

Instance 5

00000000  00 00 00 00 00 00 00 00  00 00 00 00 d0 0c 00 00  |................|
00000010  01 00 00 00 96 2a 21 81  ed 09 96 49 94 71 8d 72  |.....*!....I.q.r|
00000020  9c 8e 69 ed 01 00 00 00  00 03 00 00 20 0a 00 00  |..i......... ...|
00000030  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000050  00 00 00 00 00 00 00 00  00 00 00 00 02 02 00 00  |................|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 11 f3 87 8f  |................|
00000070  98 c9 9e 4d a0 c4 60 65  51 8c 4f 6d 02 63 03 01  |...M..`eQ.Om.c..|
00000080  80 02 00 00 5f 30 6b e1  08 00 00 00 00 00 1b 00  |...._0k.........|
00000090  05 00 00 00 ff 03 00 80  68 f6 01 00 40 71 03 00  |........h...@q..|
000000a0  24 74 02 00 00 04 00 00  01 00 02 00 00 c7 04 7f  |$t..............|
000000b0  67 ff 8f f0 09 ff 63 60  0a f1 20 60 9f 00 00 11  |g.....c`.. `....|
000000c0  20 00 20 00 00 40 00 01  00 00 80 00 04 00 00 00  | . ..@..........|
000000d0  81 08 50 20 12 cf 0e 00  5d 03 04 03 41 00 01 00  |..P ....]...A...|
000000e0  70 00 00 00 00 00 10 01  00 00 00 05 8f 30 70 5a  |p............0pZ|
000000f0  3c 00 20 00 0e 00 00 fc  01 80 c8 39 70 fe fc c7  |<. ........9p...|
00000100  3f c6 4f 7e 47 ff ff 81  db 38 70 7c 00 32 00 0f  |?.O~G....8p|.2..|
00000110  78 41 03 1e 01 c2 30 70  5a 3c 00 00 00 06 00 00  |xA....0pZ<......|
00000120  00 00 00 c0 20 00 00 00  00 00 00 00 00 00 00 00  |.... ...........|
00000130  00 00 20 20 c6 fc 00 20  00 08 00 00 00 00 00 00  |..  ... ........|
00000140  c7 83 00 03 00 c0 39 f0  00 00 00 00 40 22 18 80  |......9.....@"..|
00000150  58 00 80 06 00 00 c0 00  00 00 00 80 10 00 00 00  |X...............|
00000160  40 03 00 b8 14 01 02 00  00 c0 1f aa 8a 88 80 00  |@...............|
00000170  0a 10 01 04 00 1b a3 a0  80 20 00 02 03 fe 05 7f  |......... ......|
00000180  01 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000190  00 00 00 00 00 00 00 62  00 00 00 00 00 00 00 00  |.......b........|
000001a0  00 00 04 00 04 00 00 00  01 33 20 00 10 00 40 00  |.........3 ...@.|
000001b0  00 00 00 00 80 00 00 01  d0 ff 7f ff ff c8 77 fc  |..............w.|
000001c0  fd fd ff ff 00 00 00 09  e4 ad 90 02 00 00 00 01  |................|
000001d0  ff 0f 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001e0  00 00 72 01 10 34 00 00  00 00 00 00 00 00 00 00  |..r..4..........|
000001f0  00 00 1f 00 00 00 00 00  00 00 00 01 ff 0f 00 00  |................|
00000200  00 00 00 00 00 00 00 00  00 00 00 00 00 00 72 01  |..............r.|
00000210  00 04 00 00 00 00 00 00  00 00 00 00 00 00 1f 00  |................|
00000220  00 00 00 00 00 00 00 01  ff 0f 00 00 00 00 00 00  |................|
00000230  00 00 00 00 00 00 00 00  00 00 72 01 00 04 00 00  |..........r.....|
00000240  00 00 00 00 00 00 00 00  00 00 1f 00 00 00 00 00  |................|
00000250  00 00 00 01 ff 0f 00 00  00 00 00 00 00 00 00 00  |................|
00000260  00 00 00 00 00 00 72 01  00 04 00 00 00 00 00 00  |......r.........|
00000270  00 00 00 00 00 00 1f 00  00 00 00 00 00 00 00 01  |................|
00000280  ff 0f 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000290  00 00 72 01 10 34 00 00  00 00 00 00 00 00 00 00  |..r..4..........|
000002a0  00 00 1f 00 00 00 00 00  ad 11 18 3d ac 01 a4 01  |...........=....|
000002b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 72 01  |..............r.|
000002c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 1f 00  |................|
000002d0  00 00 00 00 00 00 00 01  ff 0f 00 00 00 00 00 00  |................|
000002e0  00 00 00 00 00 00 00 00  00 00 72 01 00 04 00 00  |..........r.....|
000002f0  00 00 00 00 00 00 00 00  00 00 1f 00 00 00 00 00  |................|
00000300  05 11 18 3b 04 01 c5 0c  00 00 00 00 00 00 00 00  |...;............|
00000310  00 00 00 00 00 00 72 01  00 00 00 00 00 00 00 00  |......r.........|
00000320  00 00 00 00 00 00 1f 00  00 00 00 00 00 00 00 01  |................|
00000330  ff 0f 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000340  00 00 72 01 10 34 00 00  00 00 00 00 00 00 00 00  |..r..4..........|
00000350  00 00 1f 00 00 00 00 00  00 00 00 01 ff 0f 00 00  |................|
00000360  00 00 00 00 00 00 00 00  00 00 00 00 00 00 72 01  |..............r.|
00000370  00 04 00 00 00 00 00 00  00 00 00 00 00 00 1f 00  |................|
00000380  00 00 00 00 00 00 00 01  ff 0f 00 00 00 00 00 00  |................|
00000390  00 00 00 00 00 00 00 00  00 00 72 01 00 04 00 00  |..........r.....|
000003a0  00 00 00 00 00 00 00 00  00 00 1f 00 00 00 00 00  |................|
000003b0  00 00 00 01 ff 0f 00 00  00 00 00 00 00 00 00 00  |................|
000003c0  00 00 00 00 00 00 72 01  00 04 00 00 00 00 00 00  |......r.........|
000003d0  00 00 00 00 00 00 1f 00  00 00 00 00 00 00 00 00  |................|
000003e0  00 00 00 00 00 00 00 00  00 00 20 2f 00 01 00 00  |.......... /....|
000003f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 02  |..............U.|
00000400  00 90 06 05 f1 00 20 00  00 00 00 44 00 00 03 1f  |...... ....D....|
00000410  24 02 cb 03 40 04 00 00  00 00 00 00 00 00 ad de  |$...@...........|
00000420  ef be ad de ef be ad de  ef be ad de ef be ad de  |................|
*
00000a70  ef be ad de ef be ad de  ef be ad de 96 2a 21 81  |.............*!.|
00000a80  ed 09 96 49 94 71 8d 72  9c 8e 69 ed 01 00 00 00  |...I.q.r..i.....|
00000a90  00 03 00 00 20 02 00 00  00 00 00 00 00 00 00 00  |.... ...........|
00000aa0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000ac0  00 00 00 00 02 02 00 00  00 00 00 00 00 00 00 00  |................|
00000ad0  00 00 00 00 11 f3 87 8f  98 c9 9e 4d a0 c4 60 65  |...........M..`e|
00000ae0  51 8c 4f 6d 01 61 03 11  80 00 00 00 00 00 00 00  |Q.Om.a..........|
00000af0  9c 3d 01 fe 00 00 00 00  34 57 50 fe 04 23 00 00  |.=......4WP..#..|
00000b00  77 5a 50 fe 02 23 00 00  68 5b 50 fe 03 23 00 00  |wZP..#..h[P..#..|
00000b10  00 5d 50 fe 05 23 00 00  9c 66 50 fe 00 23 00 00  |.]P..#...fP..#..|
00000b20  1e 91 50 fe 01 23 00 00  73 9a 50 fe 04 23 00 00  |..P..#..s.P..#..|
00000b30  b6 9d 50 fe 02 23 00 00  a7 9e 50 fe 03 23 00 00  |..P..#....P..#..|
00000b40  43 a0 50 fe 05 23 00 00  87 ab 50 fe 00 23 00 00  |C.P..#....P..#..|
00000b50  af 2d 71 ff 01 23 00 00  e9 38 71 ff 04 23 00 00  |.-q..#...8q..#..|
00000b60  2c 3c 71 ff 02 23 00 00  1d 3d 71 ff 03 23 00 00  |,<q..#...=q..#..|
00000b70  b9 3e 71 ff 05 23 00 00  6b 48 71 ff 00 23 00 00  |.>q..#..kHq..#..|
00000b80  86 2e 4c 01 01 23 00 00  db 37 4c 01 04 23 00 00  |..L..#...7L..#..|
00000b90  1e 3b 4c 01 02 23 00 00  0f 3c 4c 01 03 23 00 00  |.;L..#...<L..#..|
00000ba0  ab 3d 4c 01 05 23 00 00  ef 48 4c 01 00 23 00 00  |.=L..#...HL..#..|
00000bb0  8f fa 50 01 01 23 00 00  78 06 51 01 04 23 00 00  |..P..#..x.Q..#..|
00000bc0  bb 09 51 01 02 23 00 00  ac 0a 51 01 03 23 00 00  |..Q..#....Q..#..|
00000bd0  48 0c 51 01 05 23 00 00  89 16 51 01 00 23 00 00  |H.Q..#....Q..#..|
00000be0  f1 78 3b 02 01 23 00 00  2b 84 3b 02 04 23 00 00  |.x;..#..+.;..#..|
00000bf0  6e 87 3b 02 02 23 00 00  5f 88 3b 02 03 23 00 00  |n.;..#.._.;..#..|
00000c00  fb 89 3b 02 05 23 00 00  97 93 3b 02 00 23 00 00  |..;..#....;..#..|
00000c10  86 89 f8 03 01 23 00 00  d8 92 f8 03 04 23 00 00  |.....#.......#..|
00000c20  1b 96 f8 03 02 23 00 00  0c 97 f8 03 03 23 00 00  |.....#.......#..|
00000c30  a4 98 f8 03 05 23 00 00  e8 a3 f8 03 00 23 00 00  |.....#.......#..|
00000c40  b6 0f 01 04 01 23 00 00  f0 1a 01 04 04 23 00 00  |.....#.......#..|
00000c50  33 1e 01 04 02 23 00 00  24 1f 01 04 03 23 00 00  |3....#..$....#..|
00000c60  c0 20 01 04 05 23 00 00  51 2a 01 04 02 00 00 00  |. ...#..Q*......|
00000c70  00 20 00 01 9b a9 34 05  14 20 00 00 e7 aa 34 05  |. ....4.. ....4.|
00000c80  01 23 00 00 61 d3 5a fb  04 23 00 00 33 16 5c fb  |.#..a.Z..#..3.\.|
00000c90  02 23 00 00 24 17 5c fb  03 23 00 00 c0 18 5c fb  |.#..$.\..#....\.|
00000ca0  05 23 00 00 b3 24 5c fb  00 23 00 00 f2 79 b0 fc  |.#...$\..#...y..|
00000cb0  01 23 00 00 2c 85 b0 fc  04 23 00 00 6f 88 b0 fc  |.#..,....#..o...|
00000cc0  02 23 00 00 60 89 b0 fc  03 23 00 00 fc 8a b0 fc  |.#..`....#......|
00000cd0  05 23 00 00 98 94 b0 fc  00 23 00 00 fd 4b 50 fe  |.#.......#...KP.|
00000ce0  01 23 00 00                                       |.#..|
00000ce4

Instance 6

00000000  00 00 00 00 00 00 00 00  00 00 00 00 d0 0c 00 00  |................|
00000010  01 00 00 00 96 2a 21 81  ed 09 96 49 94 71 8d 72  |.....*!....I.q.r|
00000020  9c 8e 69 ed 01 00 00 00  00 03 00 00 20 0a 00 00  |..i......... ...|
00000030  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000050  00 00 00 00 00 00 00 00  00 00 00 00 02 02 00 00  |................|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 11 f3 87 8f  |................|
00000070  98 c9 9e 4d a0 c4 60 65  51 8c 4f 6d 02 63 03 01  |...M..`eQ.Om.c..|
00000080  80 02 00 00 bd 89 fe b8  26 00 00 00 00 00 1b 00  |........&.......|
00000090  05 00 00 00 ff 03 00 80  68 f6 01 00 64 f6 01 00  |........h...d...|
000000a0  24 74 02 00 00 00 00 00  00 00 02 00 00 c7 04 7f  |$t..............|
000000b0  67 ff 8f f0 09 ff 63 60  0a f1 20 60 9f 00 00 11  |g.....c`.. `....|
000000c0  20 00 20 00 00 40 00 01  00 00 80 00 04 00 00 00  | . ..@..........|
000000d0  81 28 50 20 12 cf 0e 00  5d 03 04 03 41 00 01 00  |.(P ....]...A...|
000000e0  70 00 00 00 00 00 10 01  00 00 00 05 8f 30 70 5a  |p............0pZ|
000000f0  3c 00 20 00 0e 00 00 fc  01 80 c8 39 70 fe fc c7  |<. ........9p...|
00000100  3f c6 4f 7e 47 ff ff 81  db 38 70 7c 00 32 00 0f  |?.O~G....8p|.2..|
00000110  78 41 03 1e 01 c2 30 70  5a 3c 00 00 00 06 00 00  |xA....0pZ<......|
00000120  00 00 00 c0 20 00 00 00  00 00 00 00 00 00 00 00  |.... ...........|
00000130  00 00 20 20 c6 fc 00 20  00 08 00 00 00 00 00 00  |..  ... ........|
00000140  c7 83 00 03 00 c0 39 f0  00 00 00 00 40 22 18 80  |......9.....@"..|
00000150  58 00 80 06 00 00 c0 00  00 00 00 80 10 00 00 00  |X...............|
00000160  40 03 00 b8 14 01 02 00  00 c0 1f aa 8a 88 80 00  |@...............|
00000170  0a 10 01 04 00 1b a3 a0  80 20 00 02 03 fe 05 7f  |......... ......|
00000180  01 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000190  00 00 00 00 00 00 00 62  00 00 00 00 00 00 00 00  |.......b........|
000001a0  00 00 04 00 04 00 00 20  41 33 20 00 90 00 40 00  |....... A3 ...@.|
000001b0  00 00 00 00 80 00 00 01  d0 ff 7f ff ff c8 77 fc  |..............w.|
000001c0  fd fd ff ff 00 00 00 09  e4 ad 90 02 00 00 00 01  |................|
000001d0  ff 0f 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001e0  00 00 72 01 10 34 00 00  00 00 00 00 00 00 00 00  |..r..4..........|
000001f0  00 00 1f 00 00 00 00 00  00 00 00 01 ff 0f 00 00  |................|
00000200  00 00 00 00 00 00 00 00  00 00 00 00 00 00 72 01  |..............r.|
00000210  00 04 00 00 00 00 00 00  00 00 00 00 00 00 1f 00  |................|
00000220  00 00 00 00 00 00 00 01  ff 0f 00 00 00 00 00 00  |................|
00000230  00 00 00 00 00 00 00 00  00 00 72 01 00 04 00 00  |..........r.....|
00000240  00 00 00 00 00 00 00 00  00 00 1f 00 00 00 00 00  |................|
00000250  00 00 00 01 ff 0f 00 00  00 00 00 00 00 00 00 00  |................|
00000260  00 00 00 00 00 00 72 01  00 04 00 00 00 00 00 00  |......r.........|
00000270  00 00 00 00 00 00 1f 00  00 00 00 00 00 00 00 01  |................|
00000280  ff 0f 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000290  00 00 72 01 10 34 00 00  00 00 00 00 00 00 00 00  |..r..4..........|
000002a0  00 00 1f 00 00 00 00 00  8f 1d 18 3d 8e 0d 79 0d  |...........=..y.|
000002b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 72 01  |..............r.|
000002c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 1f 00  |................|
000002d0  00 00 00 00 00 00 00 01  ff 0f 00 00 00 00 00 00  |................|
000002e0  00 00 00 00 00 00 00 00  00 00 72 01 00 04 00 00  |..........r.....|
000002f0  00 00 00 00 00 00 00 00  00 00 1f 00 00 00 00 00  |................|
00000300  fe 12 18 3b fd 02 bb 0f  00 00 00 00 00 00 00 00  |...;............|
00000310  00 00 00 00 00 00 72 01  00 00 00 00 00 00 00 00  |......r.........|
00000320  00 00 00 00 00 00 1f 00  00 00 00 00 00 00 00 01  |................|
00000330  ff 0f 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000340  00 00 72 01 10 34 00 00  00 00 00 00 00 00 00 00  |..r..4..........|
00000350  00 00 1f 00 00 00 00 00  00 00 00 01 ff 0f 00 00  |................|
00000360  00 00 00 00 00 00 00 00  00 00 00 00 00 00 72 01  |..............r.|
00000370  00 04 00 00 00 00 00 00  00 00 00 00 00 00 1f 00  |................|
00000380  00 00 00 00 00 00 00 01  ff 0f 00 00 00 00 00 00  |................|
00000390  00 00 00 00 00 00 00 00  00 00 72 01 00 04 00 00  |..........r.....|
000003a0  00 00 00 00 00 00 00 00  00 00 1f 00 00 00 00 00  |................|
000003b0  00 00 00 01 ff 0f 00 00  00 00 00 00 00 00 00 00  |................|
000003c0  00 00 00 00 00 00 72 01  00 04 00 00 00 00 00 00  |......r.........|
000003d0  00 00 00 00 00 00 1f 00  00 00 00 00 00 00 00 00  |................|
000003e0  00 00 00 00 00 00 00 00  00 00 20 2f 00 01 00 00  |.......... /....|
000003f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 02  |..............U.|
00000400  00 90 06 05 f1 09 20 00  00 00 00 40 00 00 03 1f  |...... ....@....|
00000410  24 02 cb 03 40 04 00 00  00 00 00 00 00 00 ad de  |$...@...........|
00000420  ef be ad de ef be ad de  ef be ad de ef be ad de  |................|
*
00000a70  ef be ad de ef be ad de  ef be ad de 96 2a 21 81  |.............*!.|
00000a80  ed 09 96 49 94 71 8d 72  9c 8e 69 ed 01 00 00 00  |...I.q.r..i.....|
00000a90  00 03 00 00 20 02 00 00  00 00 00 00 00 00 00 00  |.... ...........|
00000aa0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000ac0  00 00 00 00 02 02 00 00  00 00 00 00 00 00 00 00  |................|
00000ad0  00 00 00 00 11 f3 87 8f  98 c9 9e 4d a0 c4 60 65  |...........M..`e|
00000ae0  51 8c 4f 6d 01 61 03 11  80 00 00 00 00 00 00 00  |Q.Om.a..........|
00000af0  70 3c 01 fe 00 00 00 00  02 23 00 00 ea 3e 46 a9  |p<.......#...>F.|
00000b00  03 23 00 00 86 40 46 a9  05 23 00 00 17 4a 46 a9  |.#...@F..#...JF.|
00000b10  00 23 00 00 71 ea 0b aa  01 23 00 00 c6 f3 0b aa  |.#..q....#......|
00000b20  04 23 00 00 df 12 0c aa  02 23 00 00 d0 13 0c aa  |.#.......#......|
00000b30  03 23 00 00 6c 15 0c aa  05 23 00 00 b0 20 0c aa  |.#..l....#... ..|
00000b40  02 00 00 00 00 20 00 01  85 4a b4 aa 14 20 00 00  |..... ...J... ..|
00000b50  d1 4b b4 aa 01 23 00 00  a2 bc b1 a1 04 23 00 00  |.K...#.......#..|
00000b60  e5 bf b1 a1 02 23 00 00  d6 c0 b1 a1 03 23 00 00  |.....#.......#..|
00000b70  72 c2 b1 a1 05 23 00 00  b6 cd b1 a1 00 23 00 00  |r....#.......#..|
00000b80  d4 5a c0 a2 01 23 00 00  0e 66 c0 a2 04 23 00 00  |.Z...#...f...#..|
00000b90  51 69 c0 a2 02 23 00 00  42 6a c0 a2 03 23 00 00  |Qi...#..Bj...#..|
00000ba0  de 6b c0 a2 05 23 00 00  7a 75 c0 a2 00 23 00 00  |.k...#..zu...#..|
00000bb0  74 17 1a a4 01 23 00 00  ae 22 1a a4 04 23 00 00  |t....#..."...#..|
00000bc0  f1 25 1a a4 02 23 00 00  e2 26 1a a4 03 23 00 00  |.%...#...&...#..|
00000bd0  7e 28 1a a4 05 23 00 00  1a 32 1a a4 00 23 00 00  |~(...#...2...#..|
00000be0  cc e2 74 a4 01 23 00 00  21 ec 74 a4 04 23 00 00  |..t..#..!.t..#..|
00000bf0  64 ef 74 a4 02 23 00 00  55 f0 74 a4 03 23 00 00  |d.t..#..U.t..#..|
00000c00  f1 f1 74 a4 05 23 00 00  35 fd 74 a4 00 23 00 00  |..t..#..5.t..#..|
00000c10  28 14 bc a5 01 23 00 00  62 1f bc a5 04 23 00 00  |(....#..b....#..|
00000c20  a5 22 bc a5 02 23 00 00  96 23 bc a5 03 23 00 00  |."...#...#...#..|
00000c30  32 25 bc a5 05 23 00 00  ce 2e bc a5 00 23 00 00  |2%...#.......#..|
00000c40  59 43 c3 a6 01 23 00 00  90 4e c3 a6 04 23 00 00  |YC...#...N...#..|
00000c50  d3 51 c3 a6 02 23 00 00  c4 52 c3 a6 03 23 00 00  |.Q...#...R...#..|
00000c60  5c 54 c3 a6 05 23 00 00  f8 5d c3 a6 00 23 00 00  |\T...#...]...#..|
00000c70  6f 5b 31 a7 01 23 00 00  c4 64 31 a7 04 23 00 00  |o[1..#...d1..#..|
00000c80  07 68 31 a7 02 23 00 00  f8 68 31 a7 03 23 00 00  |.h1..#...h1..#..|
00000c90  94 6a 31 a7 05 23 00 00  d8 75 31 a7 00 23 00 00  |.j1..#...u1..#..|
00000ca0  77 99 73 a8 01 23 00 00  b1 a4 73 a8 04 23 00 00  |w.s..#....s..#..|
00000cb0  f4 a7 73 a8 02 23 00 00  e5 a8 73 a8 03 23 00 00  |..s..#....s..#..|
00000cc0  81 aa 73 a8 05 23 00 00  12 b4 73 a8 00 23 00 00  |..s..#....s..#..|
00000cd0  7c 2f 46 a9 01 23 00 00  b6 3a 46 a9 04 23 00 00  ||/F..#...:F..#..|
00000ce0  f9 3d 46 a9                                       |.=F.|
00000ce4

Instance 7

00000000  00 00 00 00 00 00 00 00  00 00 00 00 d0 0c 00 00  |................|
00000010  01 00 00 00 96 2a 21 81  ed 09 96 49 94 71 8d 72  |.....*!....I.q.r|
00000020  9c 8e 69 ed 01 00 00 00  00 03 00 00 20 0a 00 00  |..i......... ...|
00000030  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000050  00 00 00 00 00 00 00 00  00 00 00 00 02 02 00 00  |................|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 11 f3 87 8f  |................|
00000070  98 c9 9e 4d a0 c4 60 65  51 8c 4f 6d 02 63 03 01  |...M..`eQ.Om.c..|
00000080  80 02 00 00 d1 6b 3f 4c  09 00 00 00 00 00 1b 00  |.....k?L........|
00000090  05 00 00 00 ff 03 00 80  68 f6 01 00 64 15 02 00  |........h...d...|
000000a0  24 74 02 00 00 00 00 00  00 00 02 00 00 c7 04 ff  |$t..............|
000000b0  6f ff 8f f0 09 ff 63 60  0a f1 20 60 9f 00 00 11  |o.....c`.. `....|
000000c0  20 00 20 00 00 40 00 01  00 00 00 00 04 00 00 00  | . ..@..........|
000000d0  81 28 50 20 12 cf 0e 00  5d 03 00 01 00 00 00 00  |.(P ....].......|
000000e0  70 00 00 00 00 00 10 01  00 00 00 05 8f 30 70 5a  |p............0pZ|
000000f0  3c 00 20 00 0e 00 00 fc  01 80 c8 39 70 fe fc c7  |<. ........9p...|
00000100  3f c6 4f 7e 47 ff ff 81  db 38 70 7c 00 32 00 0f  |?.O~G....8p|.2..|
00000110  78 41 03 1e 01 c2 30 70  5a 3c 00 00 00 06 00 00  |xA....0pZ<......|
00000120  00 00 00 c0 20 00 00 00  00 00 00 00 00 00 00 00  |.... ...........|
00000130  00 00 20 20 c6 fc 00 20  00 08 00 00 00 00 00 00  |..  ... ........|
00000140  c7 83 00 03 00 c0 39 f0  00 00 00 00 40 22 18 80  |......9.....@"..|
00000150  58 00 80 06 00 00 c0 00  00 00 00 80 10 00 00 00  |X...............|
00000160  40 03 00 b8 14 01 02 00  00 c0 1f aa 8a 88 80 00  |@...............|
00000170  0a 10 01 04 00 1b a3 a0  80 20 00 02 03 fe 05 7f  |......... ......|
00000180  01 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000190  00 00 00 00 00 00 00 62  00 00 00 00 00 00 00 00  |.......b........|
000001a0  00 00 04 00 04 00 00 20  41 33 20 00 90 00 40 00  |....... A3 ...@.|
000001b0  00 00 00 00 80 00 00 01  d0 ff 7f ff ff c8 77 fc  |..............w.|
000001c0  fd fd ff ff 00 00 00 09  e4 ad 90 02 00 00 00 01  |................|
000001d0  ff 0f 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
000001e0  00 00 72 01 10 34 00 00  00 00 00 00 00 00 00 00  |..r..4..........|
000001f0  00 00 1f 00 00 00 00 00  00 00 00 01 ff 0f 00 00  |................|
00000200  00 00 00 00 00 00 00 00  00 00 00 00 00 00 72 01  |..............r.|
00000210  00 04 00 00 00 00 00 00  00 00 00 00 00 00 1f 00  |................|
00000220  00 00 00 00 00 00 00 01  ff 0f 00 00 00 00 00 00  |................|
00000230  00 00 00 00 00 00 00 00  00 00 72 01 00 04 00 00  |..........r.....|
00000240  00 00 00 00 00 00 00 00  00 00 1f 00 00 00 00 00  |................|
00000250  00 00 00 01 ff 0f 00 00  00 00 00 00 00 00 00 00  |................|
00000260  00 00 00 00 00 00 72 01  00 04 00 00 00 00 00 00  |......r.........|
00000270  00 00 00 00 00 00 1f 00  00 00 00 00 00 00 00 01  |................|
00000280  ff 0f 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000290  00 00 72 01 10 34 00 00  00 00 00 00 00 00 00 00  |..r..4..........|
000002a0  00 00 1f 00 00 00 00 00  ef 1c 18 3d ee 0c d9 0c  |...........=....|
000002b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 72 01  |..............r.|
000002c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 1f 00  |................|
000002d0  00 00 00 00 00 00 00 01  ff 0f 00 00 00 00 00 00  |................|
000002e0  00 00 00 00 00 00 00 00  00 00 72 01 00 04 00 00  |..........r.....|
000002f0  00 00 00 00 00 00 00 00  00 00 1f 00 00 00 00 00  |................|
00000300  84 19 18 3b 83 09 10 04  00 00 00 00 00 00 00 00  |...;............|
00000310  00 00 00 00 00 00 72 01  00 00 00 00 00 00 00 00  |......r.........|
00000320  00 00 00 00 00 00 1f 00  00 00 00 00 00 00 00 01  |................|
00000330  ff 0f 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000340  00 00 72 01 10 34 00 00  00 00 00 00 00 00 00 00  |..r..4..........|
00000350  00 00 1f 00 00 00 00 00  00 00 00 01 ff 0f 00 00  |................|
00000360  00 00 00 00 00 00 00 00  00 00 00 00 00 00 72 01  |..............r.|
00000370  00 04 00 00 00 00 00 00  00 00 00 00 00 00 1f 00  |................|
00000380  00 00 00 00 00 00 00 01  ff 0f 00 00 00 00 00 00  |................|
00000390  00 00 00 00 00 00 00 00  00 00 72 01 00 04 00 00  |..........r.....|
000003a0  00 00 00 00 00 00 00 00  00 00 1f 00 00 00 00 00  |................|
000003b0  00 00 00 01 ff 0f 00 00  00 00 00 00 00 00 00 00  |................|
000003c0  00 00 00 00 00 00 72 01  00 04 00 00 00 00 00 00  |......r.........|
000003d0  00 00 00 00 00 00 1f 00  00 00 00 00 00 00 00 00  |................|
000003e0  00 00 00 00 00 00 00 00  00 00 20 2f 00 01 00 00  |.......... /....|
000003f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 02  |..............U.|
00000400  00 90 06 05 f1 00 20 00  00 00 00 40 00 00 03 1f  |...... ....@....|
00000410  24 02 cb 03 40 04 00 00  00 00 00 00 00 00 ad de  |$...@...........|
00000420  ef be ad de ef be ad de  ef be ad de ef be ad de  |................|
*
00000a70  ef be ad de ef be ad de  ef be ad de 96 2a 21 81  |.............*!.|
00000a80  ed 09 96 49 94 71 8d 72  9c 8e 69 ed 01 00 00 00  |...I.q.r..i.....|
00000a90  00 03 00 00 20 02 00 00  00 00 00 00 00 00 00 00  |.... ...........|
00000aa0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000ac0  00 00 00 00 02 02 00 00  00 00 00 00 00 00 00 00  |................|
00000ad0  00 00 00 00 11 f3 87 8f  98 c9 9e 4d a0 c4 60 65  |...........M..`e|
00000ae0  51 8c 4f 6d 01 61 03 11  80 00 00 00 00 00 00 00  |Q.Om.a..........|
00000af0  b0 3d 01 fe 00 00 00 00  05 23 00 00 d7 3b 49 6c  |.=.......#...;Il|
00000b00  00 23 00 00 c2 6c 56 6d  01 23 00 00 17 76 56 6d  |.#...lVm.#...vVm|
00000b10  04 23 00 00 5a 79 56 6d  02 23 00 00 4b 7a 56 6d  |.#..ZyVm.#..KzVm|
00000b20  03 23 00 00 e7 7b 56 6d  05 23 00 00 2b 87 56 6d  |.#...{Vm.#..+.Vm|
00000b30  00 23 00 00 a9 58 79 6d  01 23 00 00 e3 63 79 6d  |.#...Xym.#...cym|
00000b40  04 23 00 00 26 67 79 6d  02 23 00 00 17 68 79 6d  |.#..&gym.#...hym|
00000b50  03 23 00 00 b3 69 79 6d  05 23 00 00 44 73 79 6d  |.#...iym.#..Dsym|
00000b60  00 23 00 00 df 95 d7 6e  01 23 00 00 16 a1 d7 6e  |.#.....n.#.....n|
00000b70  04 23 00 00 59 a4 d7 6e  02 23 00 00 4a a5 d7 6e  |.#..Y..n.#..J..n|
00000b80  03 23 00 00 7e af d7 6e  05 23 00 00 0f b9 d7 6e  |.#..~..n.#.....n|
00000b90  00 23 00 00 93 37 b8 6f  01 23 00 00 e8 40 b8 6f  |.#...7.o.#...@.o|
00000ba0  04 23 00 00 2b 44 b8 6f  02 23 00 00 1c 45 b8 6f  |.#..+D.o.#...E.o|
00000bb0  03 23 00 00 b8 46 b8 6f  05 23 00 00 fc 51 b8 6f  |.#...F.o.#...Q.o|
00000bc0  00 23 00 00 6d a8 e0 6f  01 23 00 00 a7 b3 e0 6f  |.#..m..o.#.....o|
00000bd0  04 23 00 00 ea b6 e0 6f  02 23 00 00 db b7 e0 6f  |.#.....o.#.....o|
00000be0  03 23 00 00 77 b9 e0 6f  05 23 00 00 fd c2 e0 6f  |.#..w..o.#.....o|
00000bf0  00 23 00 00 49 eb 44 71  01 23 00 00 83 f6 44 71  |.#..I.Dq.#....Dq|
00000c00  04 23 00 00 c6 f9 44 71  02 23 00 00 b7 fa 44 71  |.#....Dq.#....Dq|
00000c10  03 23 00 00 53 fc 44 71  05 23 00 00 e4 05 45 71  |.#..S.Dq.#....Eq|
00000c20  00 23 00 00 ed 6a 12 72  01 23 00 00 42 74 12 72  |.#...j.r.#..Bt.r|
00000c30  04 23 00 00 a5 66 13 72  02 23 00 00 96 67 13 72  |.#...f.r.#...g.r|
00000c40  03 23 00 00 32 69 13 72  05 23 00 00 76 74 13 72  |.#..2i.r.#..vt.r|
00000c50  00 23 00 00 91 78 3d 72  01 23 00 00 cb 83 3d 72  |.#...x=r.#....=r|
00000c60  04 23 00 00 0e 87 3d 72  02 23 00 00 ff 87 3d 72  |.#....=r.#....=r|
00000c70  03 23 00 00 9b 89 3d 72  05 23 00 00 2c 93 3d 72  |.#....=r.#..,.=r|
00000c80  02 00 00 00 00 20 00 01  52 75 88 72 14 20 00 00  |..... ..Ru.r. ..|
00000c90  9e 76 88 72 01 23 00 00  73 dc fe 6a 04 23 00 00  |.v.r.#..s..j.#..|
00000ca0  b6 df fe 6a 02 23 00 00  a7 e0 fe 6a 03 23 00 00  |...j.#.....j.#..|
00000cb0  43 e2 fe 6a 05 23 00 00  df eb fe 6a 00 23 00 00  |C..j.#.....j.#..|
00000cc0  a0 18 49 6c 01 23 00 00  da 23 49 6c 04 23 00 00  |..Il.#...#Il.#..|
00000cd0  b9 2f 49 6c 02 23 00 00  aa 30 49 6c 03 23 00 00  |./Il.#...0Il.#..|
00000ce0  46 32 49 6c                                       |F2Il|
00000ce4

whpenner commented 1 year ago

@Technologicat: I'm looking at the data/structures to make sure the data is here. It may take me a bit since I'll need to reformat it and make sure the data is all here.

Technologicat commented 1 year ago

@whpenner Ok. Thank you!

I uploaded the raw BERT records to Dropbox (link in the edit of the previous post) - please tell me if you need anything else.

whpenner commented 1 year ago

@Technologicat Just to let you know, I'm looking at the data, but so far it is not showing an obvious cause. I'm having to dig to look for a hint as to what may have happened. I'll let you know if I can find something.

Technologicat commented 1 year ago

@whpenner Thanks for the update. Please also let me know if I can help, for example by testing in some particular way.

Technologicat commented 1 year ago

Just in case it might help, here are four more raw BERT records. One from last week, three from today; by a quick glance, all of them are of the same kind as the previous ones (8f87f311-c998-4d9e-a0c4-6065518c4f6d).

Today's freezes happened while scrolling down a long PDF (300+ pages) in Emacs (labeled BERT9 and BERT10), and while updating a Back In Time backup (essentially rsync with hardlinks) to a USB disk (labeled BERT11).

whpenner commented 1 year ago

@Technologicat Hi. These crashlog dumps are pretty much the same as the previous ones. I don't know if this would get us more data, but could you install and enable the mcelog tool (https://mcelog.org/)? This tool would try to capture any machine checks that happen. I can't tell if there is another error that is missed because of a second error after the system hangs.

whpenner commented 1 year ago

@Technologicat Oh, can you send us a bit more of the dmesg log. We're also looking at why we don't see more of the log in dmesg.

Thanks

Technologicat commented 1 year ago

@whpenner Yes, mcelog sounds like it's worth a try. I'll try installing it manually on the weekend, to see if it still works on mainline kernel 6.2.9 (which I'm running now), or on one of the other available kernels. I don't know if the kernel configs used to produce the binaries enable MCELOG support or not.

(mcelog is no longer packaged for recent Mint/Ubuntu/Debian, seems it has been replaced by rasdaemon.)

Here is the dmesg log corresponding to the BERT dump labeled BERT8, both before and after the reboot. Before the reboot, errors only, and after the reboot, both a full log, as well as a copy showing errors only (i.e. journalctl --no-pager -b -p3).

Sorry for no complete log before the reboot - I accidentally took the data from the wrong boot when saving the full log, noticed it only now, and the original has already been auto-deleted from /var/log/.

I no longer have the other logs, but I can produce new ones by triggering the freeze again. I'll try to get a complete set or two over the weekend, including a full log before the freeze.

(However, note that there have never been any messages in dmesg from the time of the freeze - the log just suddenly ends. Remote UDP logging via the netconsole kernel module captures nothing, either. We'll have to see if the situation changes if I get mcelog working.)

Technologicat commented 1 year ago

mcelog installed fine. Running in daemon mode, with the default config.

Now just to perform some tests... I'll post again when I have the dmesg logs.

Technologicat commented 1 year ago

Here is one set of logs, before and after a freeze.

For both boots (before and after the freeze), journalctl -u mcelog is empty, except the daemon startup message Started Machine Check Exception Logging Daemon.. Searching for MCE or mce in dmesg produces no results, either.

whpenner commented 1 year ago

I suspected that may be the case.

I am not able to find anything in the logs to tell me what happened other than the crashlog output is telling me various types of resets have occurred. I suspect this is where the system hangs and you reset it or the system is resetting by itself. There isn't any data left to say why it hung/reboot. Since I'm not seeing other systems like this, I would suspect the system/hw or how FW/BIOS is responding after the hang/reboot. Your best bet may be to contact Clevo.

One other item, and maybe you are already aware of this, but I see many lwlwifi errors in the log. The references to microcode in that log is about the firmware for the wifi device and not processor microcode. That device seems to be hanging quite often. It's possible that whatever that device is doing, it could cause your system to hang/reset. That could explain some of your symptoms as well (network stops responding to pings, etc).

Note the: Apr 06 13:24:11 maia kernel: iwlwifi 0000:00:14.3: Queue 5 is stuck 44471 44650 Apr 06 13:24:11 maia kernel: iwlwifi 0000:00:14.3: Microcode SW error detected. Restarting 0x0. Apr 06 13:24:11 maia kernel: iwlwifi 0000:00:14.3: Start IWL Error Log Dump: Apr 06 13:24:11 maia kernel: iwlwifi 0000:00:14.3: Transport status: 0x0000004A, valid: 6 Apr 06 13:24:11 maia kernel: iwlwifi 0000:00:14.3: Loaded firmware version: 72.daa05125.0 so-a0-hr-b0-72.ucode

Technologicat commented 1 year ago

Ok.

Yes, I'm aware of the iwlwifi errors. The WiFi chip is an Intel AX201. I have tried running with wired ethernet, with WiFi soft-disabled from the NetworkManager GUI - no change, still freezes - but I haven't yet tried blacklisting the iwlwifi driver to completely rule it out.

If it's possible for a wifi driver to hard-freeze the entire PC - so fast that the kernel has no chance to log anything, and so hard that the whole kernel just dies, no kernel panic, and not even Alt+SysRq+REISUB responds - then I think blacklisting the driver is the next thing to try.

My hunch that the CPU microcode might have been responsible came from an internet search for the 8f87... ACPI BERT error code. I was biased toward this option, because two of the workloads that most reliably trigger the freeze both place a high multicore load on the CPU, without much (if any) network traffic.

The iwlwifi errors only seem to start appearing once the laptop has been running for a while, whereas the freeze can be reliably reproduced within minutes after booting (by running the right workload - with the chance of freezing increased if the e-cores are enabled; but it happens with only p-cores, too). Note that in the more recent log, from a freeze on April 14, the system froze without any iwlwifi errors in the log. All of this led me to think the iwlwifi errors didn't seem relevant to this particular issue.

From my internet search, I got the impression that the most common causes for an 8f87... are either a HW failure, or a component that isn't seated properly, or a BIOS bug, or a CPU microcode bug.

This is a new laptop, and the vendor I bought it from tests each individual machine they sell for two weeks before shipping, so a HW failure seems unlikely (although of course not entirely impossible).

I don't know if handling during international shipping could bump a laptop enough for the CPU or one of the DIMMs to become unseated just enough to start causing intermittent failures in one of the electrical contacts. Also seems unlikely, but not entirely impossible.

As for a BIOS bug, yes, could be a good idea for me to contact CLEVO. I don't know if they support Linux, but it never hurts to ask.

In any case, thank you for your efforts!

Technologicat commented 1 year ago

Just as a last update to this issue, it's not the iwlwifi driver causing the freezes, either. I unloaded the module (sudo modprobe -r iwlwifi), ran completely offline, and got the system to still freeze in minutes by running Cyberpunk 2077 (all CPU cores enabled, GPU at factory default full power).

So most probably the iwlwifi errors are unrelated to the freezes.

I'll go ask CLEVO next for their input - thanks for your support!

Technologicat commented 11 months ago

As a final follow-up to this: the freezing turned out to be caused a faulty RAM module that wasn't caught by memtest.

The RMA process took some time, but the faulty memory was replaced under warranty, and now everything is working fine.

whpenner commented 11 months ago

@Technologicat Thanks for following up and letting us know. I'm glad your system is now working.

mehturt commented 11 months ago

@Technologicat I have the exact same error with my new MB with N100 CPU, I'm suspecting RAM issue as well, sometimes an error is reported by memtester , other times it just freezes as soon as I run it. Out of curiosity what was the faulty RAM module that you had? Thanks.

Technologicat commented 11 months ago

@mehturt Originally, I had 32GB (2x 16GB) DDR5 4800MHz FURY Impact.

Replaced with 32GB (2x 16GB) DDR5 4800MHz Patriot Signature Line (CoreFreq reports model number PSD516G480081S).

intel / Intel-Linux-Processor-Microcode-Data-Files

System freeze on i7-12700H (06-9a-03, microcode 0x41e and 0x429) #67