Can't load firmware on UP Xtreme with 5.12 kernel

[rfvirgil]

Transferred from sof-bin to sof-docs

Board: Up Xtreme board (WHL01), BIOS version 1.9 (UPW1AM19).

I have been using 5.10-rc7 kernel with 1.6.1 sof firmware and this has been working just fine. But when I change to 5.12.0-rc8 kernel from torvalds/master the SOF firmware fails to load (see dmesg output at end of this ticket). I tried replacing the sof firmware with 1.7 from this repo, but get exactly the same problem.

I've tracked this to a kernel patch that went in between 5.10 and 5.12 that breaks firmware loading on Up Xtreme. If I revert commit https://github.com/thesofproject/linux/commit/bd8036eb1526, the firmware loads ok:

Author: Pierre-Louis Bossart <pierre-louis.bossart@linux.intel.com>
Date:   Mon Feb 8 17:18:53 2021 -0600

    ASoC: SOF: sof-pci-dev: add missing Up-Extreme quirk

    The UpExtreme board supports the community key and was missed in
    previous contributions. Add it to make sure the open firmware is
    picked by default without needing a symlink on the target.

    Fixes: 46207ca24545 ('ASoC: SOF: pci: change the default firmware path when the community key is used')
    Signed-off-by: Pierre-Louis Bossart <pierre-louis.bossart@linux.intel.com>
    Reviewed-by: Bard Liao <bard.liao@intel.com>
    Reviewed-by: Ranjani Sridharan <ranjani.sridharan@linux.intel.com>
    Link: https://lore.kernel.org/r/20210208231853.58761-1-pierre-louis.bossart@linux.intel.com
    Signed-off-by: Mark Brown <broonie@kernel.org>

diff --git a/sound/soc/sof/sof-pci-dev.c b/sound/soc/sof/sof-pci-dev.c
index 215711ac7450..9adf50b20a73 100644
--- a/sound/soc/sof/sof-pci-dev.c
+++ b/sound/soc/sof/sof-pci-dev.c
@@ -65,6 +65,13 @@ static const struct dmi_system_id community_key_platforms[] = {
                        DMI_MATCH(DMI_BOARD_NAME, "UP-APL01"),
                }
        },
+       {
+               .ident = "Up Extreme",
+               .matches = {
+                       DMI_MATCH(DMI_SYS_VENDOR, "AAEON"),
+                       DMI_MATCH(DMI_BOARD_NAME, "UP-WHL01"),
+               }

My sof firmware install has a /lib/firmware/intel/sof/community/ that matches the one from the v1.7 branch of this sof-bin github repository.

Error see if I do NOT revert the above patch:

[    2.608023] sof-audio-pci-intel-cnl 0000:00:1f.3: SoundWire enabled on CannonLake+ platform, using SOF driver
[    2.608036] sof-audio-pci-intel-cnl 0000:00:1f.3: enabling device (0000 -> 0002)
[    2.608195] sof-audio-pci-intel-cnl 0000:00:1f.3: DSP detected with PCI class/subclass/prog-if 0x040100
[    2.610378] sof-audio-pci-intel-cnl 0000:00:1f.3: use msi interrupt mode
<SNIP>
[    2.615092] sof-audio-pci-intel-cnl 0000:00:1f.3: No SoundWire machine driver found
[    2.615094] sof-audio-pci-intel-cnl 0000:00:1f.3: warning: No matching ASoC machine driver found
[    2.615095] sof-audio-pci-intel-cnl 0000:00:1f.3: Using nocodec machine driver
[    2.628716] sof-audio-pci-intel-cnl 0000:00:1f.3: Firmware info: version 1:7:0-47d07
[    2.628719] sof-audio-pci-intel-cnl 0000:00:1f.3: Firmware: ABI 3:18:1 Kernel ABI 3:18:0
[    2.628725] sof-audio-pci-intel-cnl 0000:00:1f.3: unknown sof_ext_man header type 3 size 0x30
<SNIP>
[    5.696645] sof-audio-pci-intel-cnl 0000:00:1f.3: error: cl_copy_fw: timeout HDA_DSP_SRAM_REG_ROM_STATUS read
[    5.696675] sof-audio-pci-intel-cnl 0000:00:1f.3: error: status = 0x0000002c panic = 0x00000000
[    5.696716] sof-audio-pci-intel-cnl 0000:00:1f.3: error: extended rom status:  0x80000012 0x2c 0x0 0x0 0x0 0x0 0x1811102 0x0                                                                                                         
[    5.696732] sof-audio-pci-intel-cnl 0000:00:1f.3: error: load fw failed ret: -110
[    5.696765] sof-audio-pci-intel-cnl 0000:00:1f.3: error: failed to reset DSP
[    5.696771] sof-audio-pci-intel-cnl 0000:00:1f.3: error: failed to boot DSP firmware -110
[    5.747966] sof-audio-pci-intel-cnl 0000:00:1f.3: error: hda_dsp_core_reset_enter: timeout on HDA_DSP_REG_ADSPCS read                                                                                                                
[    5.747981] sof-audio-pci-intel-cnl 0000:00:1f.3: error: dsp core reset failed: core_mask 1
[    5.748162] sof-audio-pci-intel-cnl 0000:00:1f.3: error: failed to probe DSP hardware!
[    5.748497] sof-audio-pci-intel-cnl: probe of 0000:00:1f.3 failed with error -110

[plbossart]

@rfvirgil I have also run into issues with the BIOS 1.9, and reverted to use BIOS 1.8 on another board. I assumed it was a one-off hardware problem but it looks more systematic I am afraid.

If you use the default settings, it means you are using the Intel private keys. It'll work but it will prevent you from generating your own firmware (assuming you have access to tools).

The best way to test this hypothesis is to make a backup of your /lib/firmware/intel/sof setup, and copy sof-cnl.ri into community/sof-cnl.ri. if this fails again then indeed it's a firmware signature and therefore BIOS issue.

[rfvirgil]

I don't need to build my own firmware so that isn't a problem. But I do need some features that were added in BIOS 1.9 so I can't downgrade to 1.8.

I've confirmed that WITH your original patch, if I do cp /lib/firmware/intel/sof/intel-signed/sof-cnl.ri /lib/firmware/intel/sof/community the firmware loads.

[plbossart]

@rfvirgil I'll report the issue to AAEON. thanks!

[marc-hb]

@plbossart did you report that problem? Then we can close this as "transferred"

[plbossart]

I did report the issue but I don't recall having seen an update.

[marc-hb]

I had a similar issue with BIOS 1.9 but that was because I was not updating the ME part(s) of the BIOS. Paying attention to the output of the BIOS upgrade script showed the partly failed upgrade. The problem was solved and I could load community signed SOF after following these steps EXACTLY:

• Go to Boot menu, then change the Boot Option 1 to select "UEFI AP: UEFI: Built-in EFI Shell'. this is needed to go directly to the EFI shell in step 8.
• Go to Main -> CRB Setup -> CRB Advanced -> PCH-FW Configuration -> Firmware Update Configuration
• Change Option ; "Me FW Re-Flash" from Disabled to Enabled . Warning: this option seems not persistent!
• F4: Save and Exit, and let the device boot to EFI shell
• fs0:
• GO_Entire.nsh
• After all process completed , system must power off to have the changes take effect ! The system may reboot more than once, don't interrupt it.
• Pay attention to the upgrade progress and make sure the ME gets updated.

[marc-hb]

I did report the issue but I don't recall having seen an update.

@plbossart can we expect AAEON BIOS to always be configured to check the community key by default? Is that some official commitment they made?

BTW I suspect it may be possible to allow more than one key but not sure about that.

I've tracked this to a kernel patch that went in between 5.10 and 5.12 that breaks firmware loading on Up Xtreme. If I revert commit https://github.com/thesofproject/linux/commit/bd8036eb1526 [...] But I do need some features that were added in BIOS 1.9 so I can't downgrade to 1.8. [...] I've confirmed that WITH your original patch, if I do cp /lib/firmware/intel/sof/intel-signed/sof-cnl.ri /lib/firmware/intel/sof/community the firmware loads.

@rfvirgil commit v5.12-rc2~155^2~1^2~1^2-gbd8036eb1526 changed which /lib/firmware/ subdirectory the SOF firmware is loaded from but it cannot and does not change which key and signature are allowed. The authentication does not care and does not even know which subdirectory the SOF firmware file came from.

So this means your BIOS has been expecting intel-signed firmware the whole time and copying the intel-signed file to the community directory as done above is a perfect workaround as long as you are not interested in running any random community-signed firmware. Are you? Conversely, is it important for security reasons to run only official, intel-signed SOF firmware? Or do you just want audio to work with any signature?

The snd_sof_pci and snd_sof_acpi modules have a fw_path parameter that should provide a different workaround if you prefer (untested), you can use that in /etc/modprobe.d/ and it will persist across SOF firmware upgrades.

Also, the easiest way to check which subdirectory the kernel loads from is to temporarily rename sof-cnl.ri in ALL subdirectories and find the error message in the logs.

[plbossart]

@marc-hb AAEON have been very supportive of SOF developments, but the BIOS is typically modified to support the SOF community key after the initial bring-up. We can safely assume that the authentication will be enabled at some point.

[marc-hb]

After a short chat with @plbossart and some additional testing on my board, great news: I confirm that my board can boot firmware signed with EITHER the official intel key OR the community key. Both work!

So @rfvirgil I suspect you simply did not opt-in to the ME part of the 1.9 upgrade. Opting in is a bit tricky, it requires following the very specific steps above in exact sequence and paying attention to the output of the update program.

Of course do NOT opt-in if you want to make sure your board keeps booting official, intel-signed firmware only and nothing else. Then you unfortunately need one of the workarounds I mentioned above because kernels 5.12 and above will always try to load community/sof-cnl.ri (whatever is actually in this file)

@plbossart , if my guess is correct wrt. to @rfvirgil 's upgrade then there is no 1.9 bug on the AAEON side.

Unless someone objects in the next week or two I will close this bug.

[plbossart]

@plbossart , if my guess is correct wrt. to @rfvirgil 's upgrade then there is no 1.9 bug on the AAEON side.

I don't know about that one. We had issues with sof-test reporting authentication issues during suspend-resume stress tests, and for that reason we all reverted to 1.8. I think if you use 1.9 for development things are probably fine. If you use 1.9 for non-regressions/validations your mileage may vary.

[marc-hb]

We had issues with sof-test reporting authentication issues during suspend-resume stress tests,

Very useful, thanks for sharing. Seems like a very different issue though (and probably much harder to fix)

[lgirdwood]

@marc-hb it's probably worth while posting your steps above on sof-docs since it's tried and tested sequence. :)

[marc-hb]

It's tried and tested on that specific board with that specific BIOS. Do we really want sof-docs to get into product-specific BIOS documentation?

[lgirdwood]

It's tried and tested on that specific board with that specific BIOS. Do we really want sof-docs to get into product-specific BIOS documentation?

Yes - as this is and Chromebooks are the recommended developer devices.

[marc-hb]

off-topic: Chromebook help added in https://github.com/thesofproject/sof-docs/pull/379