Some P8/P9 platforms need to sign by default(STB verification fails due to unsigned images).

[pridhiviraj]

/ # cat /sys/firmware/opal/msglog | grep -i STB
[   83.093364969,6] MEM: parsing reserved memory from node /ibm,hostboot/reserved-memory
[   86.447190140,5] STB: Found ibm,secureboot-v1
[   86.447255337,5] STB: secure mode off
[   86.448263983,6] STB: Found CVC @ 305b90e0-305bd0df
[   86.448265371,6] STB: Found CVC-sha512 @ 305b9100, version=1
[   86.448266765,6] STB: Found CVC-verify @ 305b9110, version=1
[   86.448270922,5] STB: trusted mode off
[   86.494176207,0] STB: CAPP verification FAILED. log=0xffffffffffff8420
[   96.184210794,0] STB: BOOTKERNEL verification FAILED. log=0xffffffffffff8420
/ # uname  -a
Linux skiroot 4.14.13-openpower1 #2 SMP Fri Feb 2 12:35:56 IST 2018 ppc64le GNU/Linux
/ # cat /etc/os-release 
NAME=Buildroot
VERSION=2017.11-00005-g65679be
ID=buildroot
VERSION_ID=2017.11
PRETTY_NAME="Buildroot 2017.11"
/ # ipmitool  fru print 47
 Device not present (Requested sensor, data, or record not found)
/ # ipmitool  fru print 43
 Product Name          : OpenPOWER Firmware
 Product Version       : open-power-habanero-v1.20-121-g38cfc32
 Product Extra         :    buildroot-2017.11-5-g65679be
 Product Extra         :    skiboot-v5.9-240-g081882690163
 Product Extra         :    hostboot-d3025f5
 Product Extra         :    linux-4.14.13-openpower1-p2027601
 Product Extra         :    petitboot-v1.6.6-p451cb3f
 Product Extra         :    machine-xml-6a78496-pd2df5b1
 Product Extra         :    occ-28
/ # 

[cclaudio]

You seem to be running an old skiboot version. The opal log says that it found ibm,secureboot-v1, but latest is v2.

Skiboot doesn't expect images to be signed if secure-mode is OFF. Talking to @stewart-ibm we decided that skiboot should always try to verify images before execution, but the boot should be halted only if secure mode is ON and the verification fails.

[pridhiviraj]

@cclaudio The skiboot is pretty latest v5.9-240-g081882690163. So FYI, this issue is in P8 systems where i guess it will be ibm,secureboot-v1 not the v2. And also as you said, by default we are now verifying the images, so during the process of verifying it, it looks for signed image, but the PNOR still an unsigned PNOR image and hence the verification fails.

[pridhiviraj]

Even on a P9 system(p9dsu.pnor which built from upstream op-build in local mode) observed the verify fails when we flash a unsigned PNOR image.

[root@ltc-boston-xxx ~]# cat /sys/firmware/opal/msglog | grep  -i STB
[  182.235611941,6] MEM: parsing reserved memory from node /ibm,hostboot/reserved-memory
[  186.076518933,0] STB: VERSION NOT VERIFIED, invalid param. buf=0x305eea20, len=4096 key-hash=0x0 hash-size=0
[  188.076375981,5] STB: Found ibm,secureboot-v2
[  188.080511860,5] STB: secure mode off
[  188.082641745,6] STB: Found CVC @ 203ffd260000-203ffd26ffff
[  188.082643550,6] STB: Found CVC-sha512 @ 203ffd260040, version=1
[  188.082645145,6] STB: Found CVC-verify @ 203ffd260050, version=1
[  188.082647617,5] STB: trusted mode off
[  188.299482128,0] STB: IMA_CATALOG verification FAILED. log=0xffffffffffff8160
[  188.430889792,0] STB: CAPP verification FAILED. log=0xffffffffffff8160
[  194.321807659,0] STB: BOOTKERNEL verification FAILED. log=0xffffffffffff8160
[root@ltc-boston-xxx ~]# 
[root@ltc-boston-xxx ~]# ipmitool  fru  print 47
 Product Name          : OpenPOWER Firmware
 Product Version       : open-power-p9dsu-v1.20-123-g76d2ca9-dirty
 Product Extra         :    buildroot-2017.11-5-g65679be
 Product Extra         :    skiboot-v5.9-240-g081882690163
 Product Extra         :    hostboot-9bfb201
 Product Extra         :    linux-4.14.16-openpower1-p2735a6b
 Product Extra         :    petitboot-v1.6.6-p451cb3f
 Product Extra         :    machine-xml-fb5f933
 Product Extra         :    occ-577915f
[root@ltc-boston-xxx ~]# 

[pridhiviraj]

1. Build a unsigned PNOR image.
2. Flash it on to the system.
3. Check the DT properties of STB
 / # cd /proc/device-tree/ibm,secureboot/
/sys/firmware/devicetree/base/ibm,secureboot # lsprop  .
compatible       "ibm,secureboot-v2"
hw-key-hash-size 00000040 (64)
hw-key-hash      00000000 00000000 00000000 00000000
         00000000 00000000 00000000 00000000
         00000000 00000000 00000000 00000000
         00000000 00000000 00000000 00000000
phandle          0000009e (158)
name             "ibm,secureboot"
/sys/firmware/devicetree/base/ibm,secureboot # 

4.When Secureboot is OFF:
====================
/ # 
/ # cat /sys/firmware/opal/msglog | grep -i STB
[  141.144150234,6] MEM: parsing reserved memory from node /ibm,hostboot/reserved-memory
[  144.472521463,0] STB: VERSION NOT VERIFIED, invalid param. buf=0x305eea20, len=4096 key-hash=0x0 hash-size=0
[  146.473282041,5] STB: Found ibm,secureboot-v2
[  146.477418883,5] STB: secure mode off
[  146.479549335,6] STB: Found CVC @ 203ffd260000-203ffd26ffff
[  146.479551234,6] STB: Found CVC-sha512 @ 203ffd260040, version=1
[  146.479552953,6] STB: Found CVC-verify @ 203ffd260050, version=1
[  146.479555421,5] STB: trusted mode off
[  147.184385396,0] STB: IMA_CATALOG verification FAILED. log=0xffffffffffff8160
[  147.315811917,0] STB: CAPP verification FAILED. log=0xffffffffffff8160
[  153.206557808,0] STB: BOOTKERNEL verification FAILED. log=0xffffffffffff8160
/ # 

Boot passes with just Verification fails 

5. When Secureboot is ON:
====================
[   62.275847941,5] STB: Found ibm,secureboot-v2
[   62.275855854,7] LPC-MBOX: Sending BMC interrupt
[   62.276664278,6] BT: seq 0x05 netfn 0x06 cmd 0x06: IPMI MSG done
[   62.276668642,6] BT: seq 0x06 netfn 0x06 cmd 0x2e: Message sent to host
[   62.286597908,5] STB: secure mode on (FORCED by nvram)
[   62.288742457,5] STB: trusted mode off
[   62.290860087,5] OPAL: Using OPAL UART console
[   62.292995902,7] LPC: Routing irq 4, policy: 0 (r=1)
[   62.292996977,7] LPC: SerIRQ 4 using route 2 targetted at OPAL
[   62.298712708,5] SLW: Configuring self-restore for HRMOR
[   62.301554322,5] SLW: Configuring self-restore for HRMOR
[   62.304356266,5] SLW: Enabling: stop0_lite
[   62.306476237,5] SLW: Enabling: stop0
[   62.307901047,5] SLW: Enabling: stop1_lite
[   62.310029660,5] SLW: Enabling: stop1
[   62.312157981,5] SLW: Enabling: stop2_lite
[   62.314287273,5] SLW: Enabling: stop2
[   62.316765000,5] VAS: Initialized chip 0
[   62.319167365,5] VAS: Initialized chip 8
[   62.320797403,5] Found ibm,power9-nx
[   62.322919575,5] Found ibm,power9-nx
[   62.397897430,0] STB: IMA_CATALOG verification FAILED. log=0xffffffffffff8160
[   62.400927445,0] STB: secure mode enforced, aborting.
[   62.403765717,0] Aborting!
CPU 0000 Backtrace:
 S: 0000000031c039c0 R: 00000000300136d8   .backtrace+0x48
 S: 0000000031c03a60 R: 0000000030019ed0   ._abort+0x4c
 S: 0000000031c03ae0 R: 000000003009cde4   .secureboot_enforce+0x3c
 S: 0000000031c03b50 R: 000000003009d204   .secureboot_verify+0xc0
 S: 0000000031c03bf0 R: 000000003002ec6c   .flash_load_resources+0x680
 S: 0000000031c03d40 R: 0000000030018088   .cpu_process_jobs+0xdc
 S: 0000000031c03e00 R: 0000000030014d38   .__secondary_cpu_entry+0x44
 S: 0000000031c03e80 R: 0000000030014d8c   .secondary_cpu_entry+0x34
 S: 0000000031c03f00 R: 0000000030002748   secondary_wait+0x8c
[   63.072968605,4] IPMI: Dropped eSEL: BMC code is buggy/missing

Verification fails and aborted the boot.

@cclaudio @stewart-ibm @hellerda So what will be the expected behaviour in this case when flashing unsigned images(When both SB is ON and OFF).?

[pridhiviraj]

I didn't see the status/return code log=0xffffffffffff8160 in https://github.com/open-power/hostboot/blob/master/src/include/securerom/status_codes.H

[cclaudio]

log=0xffffffffffff8160means that the hardware key hash test failed. At runtime, the secure ROM code reads the three hw keys form the container, concatenate them and calculate a hash. The hash is then compared to the hw-key-hash trusted by the platform owner, which was installed previously in the system.

At runtime the return code is generated by applying binary AND operations. log=0xffffffffffff8160 = ERROR_EVENT & ROM_VERIFY & HW_KEY_HASH_TEST

[cclaudio]

In ibm,secureboot-v2 skiboot always try to verify images retrieved from PNOR, but enforce only when secure-mode is ON.

4 and 5 are working as expected for me.

In 4, secure mode is OFF, then the system should boot even if a PNOR partition verification fails.

In 5, secure mode is ON and the system is running an unsigned PNOR. So it expected that skiboot halts the boot when the first PNOR partition is retrieved (IMA_CATALOG in this case) because the verification will fail.

AFAIK hostboot will populate the secure boot entries in the HDAT only if the system is booting in secure mode. In 5, secure mode is forced only in skiboot (by NVRAM), so I think hostboot is not booting in secure mode. That would explain why the hw-key-hash is zeroed in the device tree.

[pridhiviraj]

@cclaudio Thanks. Agree 4 and 5 are expected. but in case of 4 need message log level should be ERROR or WARNING, as EMERGENCY level should only for boot abort or boot crashes. As we are proceeding for booting in case of 4 the log level should be reduced.

[pridhiviraj]

@cclaudio Surprisingly, even though the secure mode is OFF, the HDAT populated all secureboot entries. This output verified with a signed PNOR image(imprint).

[root@ltc-boston-xxx ~]# cat /sys/firmware/opal/msglog | grep -i STB
[  183.217993406,6] MEM: parsing reserved memory from node /ibm,hostboot/reserved-memory
[  187.020499236,0] STB: VERSION NOT VERIFIED, invalid param. buf=0x305ed930, len=4096 key-hash=0x0 hash-size=0
[  187.137075516,5] STB: Found ibm,secureboot-v2
[  187.147912534,5] STB: secure mode off
[  187.149338688,6] STB: Found CVC @ 203ffd280000-203ffd28ffff
[  187.149340371,6] STB: Found CVC-sha512 @ 203ffd280040, version=1
[  187.149342136,6] STB: Found CVC-verify @ 203ffd280050, version=1
[  187.149345683,5] STB: trusted mode off
[  187.272604318,6] STB: IMA_CATALOG verified
[  187.414517444,6] STB: CAPP verified
[  195.045360795,6] STB: BOOTKERNEL verified
[root@ltc-boston-xxx ~]# cd /proc/device-tree/ibm,secureboot/
[root@ltc-boston-xxx ibm,secureboot]# lsprop .
compatible       "ibm,secureboot-v2"
hw-key-hash-size 00000040 (64)
hw-key-hash      40d487ff 7380ed6a d54775d5 795fea0d
         e2f541fe a9db06b8 466a42a3 20e65f75
         b4866546 0017d907 515dc2a5 f9fc5095
         4d6ee0c9 b67d219d fb708535 1d01d6d1
phandle          0000009e (158)
name             "ibm,secureboot"
[root@ltc-boston-xxx ibm,secureboot]# 

[hellerda]

It seems like the question is: for systems with secure mode disabled (by jumper), do the containers need to be:

1. Signed, even though the secure boot checks are not enforced?
2. Unsigned, but still need the 4k container header? (i.e. mock container)
3. Should not have the 4k header at all?

[dcrowell77]

Adding @bofferdn to the conversation, he's our secureboot guy

[pridhiviraj]

@hellerda That looks great, you have formed the problem statement is clear now. We need a clear picture on how the build side(op-build) and system boot side, what images op-build by default need to build and which user can flash and boot properly on both secure mode enabled and disabled systems(On both P8 and P9). This needs to have a look at op-build, PNOR layout, and all firmware layers secure boot implementation.

[ghost]

Our philosophy in skiboot has been "always attempt to verify the container and signature, even if no TPM and secureboot=off" on the idea that we'd at least be able to warn about potential problems. Realistically, the only valid case for doing that is if you're a developer or doing something you probably shouldn't - certainly not something that should be normal in production environments.

Of course, for P8, Hostboot was fairly intolerant of a secure boot header being there where it didn't expect one, while on P9 it seems like things behave as I'd expect them... although I haven't looked too closely.

I'm inclined to say we should start signing everything by default, with some workaround for P8?

Or we change the priority of the message in skiboot on P8 with securemode=off....

@hellerda what are your thoughts?

[ghost]

Is there any reason not to enable the SECUREBOOT hostboot option by default on p9?

[hellerda]

I'll comment on the philosophy, but I don't want to lose sight of the bug here, as it may have some simple fix that can unblock the the test. So on this bug specifically:

Claudio: a) Is there an issue here with hostboot not populating the device tree with the hash, independent of whether or not the container is signed? b) Does Pridhiviraj have the right code level? Should he be seeing v1 or v2 in the device tree?

On the philosophy: Consistency is always better. To support just the presence of a header it takes no more than the find the magic number and skip 4k, if there's no secure boot supported (or required) for that module. So ideally, all verification code should tolerate the presence of the header, if the module being loaded is ever secure booted, anywhere. That said, we don't need to fix anything that's not broken, esp. on P8.

[bofferdn]

for P9, at minimum, even when you compile secure boot out, the build process is putting in mock headers equal to the normal secure header in every way -except- the signatures are empty. This facilitates more or less one code path in the firmware, more robust and structured ability to do basic verification of content (like is this component really the OCC LID, etc.), etc. that you would conceivably want even if you don't care to boot secure. Code is free to simply skip final signature verification in that case, but otherwise everything looks "the same".

[bofferdn]

P8 since it's not being heavily developed, is definitely way more varied in its behavior. There are no mock headers in non-secure compile for example.

[pridhiviraj]

https://lists.ozlabs.org/pipermail/skiboot/2018-March/010577.html Patch merged upstream to reduce the severity of secure verify failure messages when there is no secureboot enabled.

[hellerda]

Pridhiviraj, can you summarize where we are with this bug? Can you list the cases where PNOR components are missing a header and the the code expecting it, vs. the cases where the header is present but the code is not handling it properly?

[pridhiviraj]

@hellerda Yeah, as of now what i observed is on all P9 system PNOR should sign by default. Witherspoon systems already signed by default and for boston not yet, fix for these systems @jimsmc already created a PR here https://github.com/open-power/op-build/pull/1988. And for all p8 systems signing is not by default, i am not sure what we can do with these systems. And regarding headers i think in code we are handling properly. @stewart-ibm Can have right answer for this in skiboot how we are handling it.