Boot only the primary cpu?

[gjinhui]

I try to boot the linux kenrel by EFI and grub2, but i find that only one cpu is online after linux kernel started. Has anyone had the same problem？

The dmesg is as follow:

...
[    0.002568] rcu: Hierarchical SRCU implementation.
[    0.003508] Remapping and enabling EFI services.
[    0.004285] smp: Bringing up secondary CPUs ...
[    5.090731] CPU1: failed to come online
[    5.090744] CPU1: failed in unknown state : 0x0
[   10.213500] CPU2: failed to come online
[   10.213512] CPU2: failed in unknown state : 0x0
[   15.336265] CPU3: failed to come online
[   15.336278] CPU3: failed in unknown state : 0x0
[   15.336306] smp: Brought up 1 node, 1 CPU
[   15.336315] SMP: Total of 1 processors activated.
...

The iomem is as follow:

root@ubuntu:/home/ubuntu# cat /proc/iomem 
00000000-001fffff : reserved
00200000-3395ffff : System RAM
  29000000-2cffffff : reserved
  2d2e0000-2d2ecfff : reserved
  2d4e0000-2d4e0fff : reserved
  2f160000-2f16ffff : reserved
  2f173000-31968fff : reserved
33960000-339bffff : reserved
339c0000-339c2fff : System RAM
  339c0000-339c0fff : reserved
339c3000-33b1ffff : reserved
33b20000-36ffffff : System RAM
  33b20000-33b2ffff : reserved
  3555e000-3555efff : reserved
37000000-373dffff : reserved
373e0000-3b3fffff : System RAM
  37400000-3b3fffff : reserved
40000000-fbffffff : System RAM
  c3d30000-c506ffff : Kernel code
  c5070000-c560ffff : reserved
  c5610000-c598ffff : Kernel data
fd500000-fd50930f : fd500000.pcie pcie@7d500000
...

I try to print cpu-release-addr when smp init. The cpu-release-addr is the same as defined in dtb. It seems that the cpu can't get rid of wfe or the cpu not in wfe?

root@ubuntu:~/RPI_EFI# dmesg |grep test
[    0.001909] [test] release_addr: ffff80000800d0e0, 00000000000000e0
[    0.001929] [test] release_addr: ffff8000080150e8, 00000000000000e8
[    0.001942] [test] release_addr: ffff8000080210f0, 00000000000000f0
[    0.004969] [test] write_pen_release cpu 1
[    5.091009] [test] write_pen_release cpu 2
[   10.213496] [test] write_pen_release cpu 3

root@ubuntu:~# cat dts |grep release
                        cpu-release-addr = <0x00 0xd8>;
                        cpu-release-addr = <0x00 0xe0>;
                        cpu-release-addr = <0x00 0xe8>;
                        cpu-release-addr = <0x00 0xf0>;

[TheMindVirus]

Which "kenrel".img are you using? There is normally an id check for processor 1 in mpidr_el1 at a very early assembly stage. If it is processor 1 it will continue to boot to linux. If it is false then the other processors are sent to a wait for event wfe loop. If the check fails and the co-processor is not working then all processors will either try to boot linux or all will be sent to loop. Either way, in the 3rd case the user would experience degraded multi-threaded performance. Maybe try a different board?

[gjinhui]

我已收到您的邮件，稍后我会仔细阅读。

[gjinhui]

Which "kenrel".img are you using?

Linux kernel 5.15 ubuntu 22.04.1 server

[gjinhui]

Maybe try a different board?

If i don't boot with EFI and grub2, it call run well.

[gjinhui]

There is normally an id check for processor 1 in mpidr_el1 at a very early assembly stage.

Do you mean the check is done in the linux kernel?

If it is processor 1 it will continue to boot to linux. If it is false then the other processors are sent to a wait for event wfe loop. If the check fails and the co-processor is not working then all processors will either try to boot linux or all will be sent to loop. Either way, in the 3rd case the user would experience degraded multi-threaded performance.

I wonder if RPI_EFI.fd will run on all the CPUs? Will the other processors sent to a wait for event wfe loop by RPI_EFI.fd except for the boot one?

[TheMindVirus]

https://github.com/TheMindVirus/PiX-iES/tree/pi4-smp-wfe-sev I've just tested it; the cores are definitely there. Either Linux or Pi4 UEFI is missing a sev instruction somewhere along the line. Each one can also be made to run their own thing entirely but with incoherent registers and exclusively shared peripherals. Not quite sure how I did that in one night but there it is...

[gjinhui]

https://github.com/TheMindVirus/PiX-iES/tree/pi4-smp-wfe-sev I've just tested it; the cores are definitely there. Either Linux or Pi4 UEFI is missing a sev instruction somewhere along the line. Each one can also be made to run their own thing entirely but with incoherent registers and exclusively shared peripherals. Not quite sure how I did that in one night but there it is...

Thanks for your hard work.

[gjinhui]

Either Linux or Pi4 UEFI is missing a sev instruction somewhere along the line.

Linux kernel will call sev during the startup period. But i am not sure Pi4 UEFI will do it or not. (start4.elf -> RPI_EFI.fd -> grubaa64.efi -> linux kernel)

static int smp_spin_table_cpu_prepare(unsigned int cpu)
{
        __le64 __iomem *release_addr;
        phys_addr_t pa_holding_pen = __pa_symbol(function_nocfi(secondary_holding_pen));

        if (!cpu_release_addr[cpu])
                return -ENODEV;

        /*
         * The cpu-release-addr may or may not be inside the linear mapping.
         * As ioremap_cache will either give us a new mapping or reuse the
         * existing linear mapping, we can use it to cover both cases. In
         * either case the memory will be MT_NORMAL.
         */
        release_addr = ioremap_cache(cpu_release_addr[cpu],
                                     sizeof(*release_addr));
        if (!release_addr)
                return -ENOMEM;

        /*
         * We write the release address as LE regardless of the native
         * endianness of the kernel. Therefore, any boot-loaders that
         * read this address need to convert this address to the
         * boot-loader's endianness before jumping. This is mandated by
         * the boot protocol.
         */
        writeq_relaxed(pa_holding_pen, release_addr);
        dcache_clean_inval_poc((__force unsigned long)release_addr,
                            (__force unsigned long)release_addr +
                                    sizeof(*release_addr));

        /*
         * Send an event to wake up the secondary CPU.
         */
        sev();

        iounmap(release_addr);

        return 0;
}

#define sev()           asm volatile("sev" : : : "memory")

[gjinhui]

The problem is solved. RPI_EFI.fd can work well. (1) It should not use spin-table to initialize the secondary CPUs. Because RPI_EFI.fd would run in EL3 and use sev to wakeup the secondary CPUs. (2) RPI_EFI.fd would modify the device tree to use psci to boot the linux kernel. (3) grub2 would load dtb again by using command devicetree. It would cover the dtb passed by RPI_EFI.fd. Thus，grub2 should not load the dtb. Or it can load the dtb which is modifed to use psci by dtc.

[jlinton]

Ok, sounds like you changed the DT to one that didn't export PSCI and it caused problems.

Closing.

[gjinhui]

我已收到您的邮件，稍后我会仔细阅读。