loongarch support

[nickdesaulniers]

cc @xen0n

@xen0n sent patches for loongarch enablement and mentioned qemu support.

@xen0n can you please share the qemu command line invocation you use for boot testing? That will help us wire up support for boot testing the kernel image.

Do you happen to know if buildroot supports loongarch?

[groeck]

On Fri, Jun 23, 2023 at 09:49:57AM -0700, Nick Desaulniers wrote:

cc @xen0n

@xen0n [sent patches for loongarch @./) and [mentioned qemu @./).

@xen0n can you please share the qemu command line invocation you use for boot testing? That will help us wire up support for boot testing the kernel image.

Examples:

qemu-system-loongarch64 -M virt -cpu la464 \ -kernel arch/loongarch/boot/vmlinuz.efi -smp 2 -no-reboot -m 4G \ -initrd rootfs.cpio \ -bios firmware/QEMU_EFI-loongarch64.fd \ --append "rdinit=/sbin/init console=ttyS0,115200" \ -nographic -serial stdio -monitor none

qemu-system-loongarch64 -M virt -cpu la464 \ -kernel arch/loongarch/boot/vmlinuz.efi -smp 2 -no-reboot -m 4G \ -bios firmware/QEMU_EFI-loongarch64.fd \ -snapshot \ -device nvme,serial=foo,drive=d0 \ -drive file=rootfs.ext2,if=none,format=raw,id=d0 \ --append "root=/dev/nvme0n1 rootwait console=ttyS0,115200" \ -nographic -serial stdio -monitor none

You'll need a working EFI bios.

Do you happen to know if buildroot supports loongarch?

I added support for it in my buildroot repository.

Guenter

[xen0n]

Thanks for the prompt followup after the patch series!

My QEMU invocation FYI:

qemu-system-loongarch64 \
        -m 4G \
        -smp 4 \
        -bios ./QEMU_EFI_7.2.fd \
        -nographic \
        -net nic -net user \
        -hda ./rw.qcow2 \
        -kernel ./vmlinuz.efi -initrd ./initramfs-linux.img \
        -append 'root=UUID=fbaa5a22-85ff-4127-82fb-efb7eba03595 rw rootfstype=ext4 loglevel=7 console=ttyS0,115200'

This is a simplistic setup with @yetist's Arch Linux port. It could be useful but given we build our own sysroots I guess we only have to take the firmware from there.

The QEMU firmware can be built from 100% open-source code, and I've previously confirmed it to be working, so we could do that as well if anyone's doubting whether we can legally just grab that prebuilt firmware blob. Last time I asked they're working on integrating the firmware into the upstream QEMU build process too; it seems they've been busy doing other things but eventually this part should become as painless as other arches.

Now it's a bit late here in UTC+8, so I'd have to go AFK soon, but I guess the info should be enough to get something together. Thanks again for the heads-up!

[xen0n]

Update: apparently the firmware is officially hosted at https://github.com/loongson/Firmware/tree/main/LoongArchVirtMachine -- maintained directly by the Loongson firmware team (who are very helpful and responsive). This could be more accessible (I'm not sure about connectivity to the Loong Arch Linux mirrors outside of China -- it's not globally mirrored yet) and having an issue tracker is definitely better.

[groeck]

On 6/23/23 10:46, WÁNG Xuěruì wrote:

Thanks for the prompt followup after the patch series!

My QEMU invocation FYI:

qemu-system-loongarch64 \ -m 4G \ -smp 4 \ -bios ./QEMU_EFI_7.2.fd \ -nographic \ -net nic -net user \ -hda ./rw.qcow2 \ -kernel ./vmlinuz.efi -initrd ./initramfs-linux.img \ -append'root=UUID=fbaa5a22-85ff-4127-82fb-efb7eba03595 rw rootfstype=ext4 loglevel=7 console=ttyS0,115200'

This is a simplistic setup https://mirrors.wsyu.edu.cn/loongarch/archlinux/images/README.html with @yetist https://github.com/yetist's Arch Linux port. It could be useful but given we build our own sysroots I guess we only have to take the firmware from there.

The QEMU firmware can be built from 100% open-source code https://github.com/tianocore/edk2-platforms/tree/master/Platform/Loongson/LoongArchQemuPkg, and I've previously confirmed it to be working, so we could do that as well if anyone's doubting whether we can legally just grab that prebuilt firmware blob. Last time I asked they're working on integrating @.***/T/#u> the firmware into the upstream QEMU build process too; it seems they've been busy doing other things but eventually this part should become as painless as other arches.

I built mine from edk2 @.***:tianocore/edk2.git), but it took me a while to find a working version. I ended up with edk2-stable202211. edk2-stable202302 does not work, and the master branch did not work either when I tried. I did not try edk2-stable202305.

Guenter

[nathanchance]

I have this tentatively wired up:

http://github.com/nathanchance/boot-utils/commits/loongarch

However, I see a crash when running /init when booting an LLVM=1 kernel:

$ ./boot-qemu.py -k $TMP_BUILD_FOLDER/linux-next

QEMU location: /usr/sbin

QEMU version: QEMU emulator version 8.0.2
$ timeout --foreground 3m stdbuf -eL -oL /usr/sbin/qemu-system-loongarch64 -display none -nodefaults -M virt -cpu la464 -bios $PWD/images/loongarch/edk2-loongarch64-code.fd -no-reboot -append console=ttyS0,115200 -kernel $KBUILD/arch/loongarch/boot/vmlinuz.efi -initrd $PWD/images/loongarch/rootfs.cpio -m 4G -smp 2 -serial mon:stdio
...
[    0.000000] Linux version 6.4.0-rc7-next-20230626+ (nathan@dev-arch.thelio-3990X) (ClangBuiltLinux clang version 17.0.0 (https://github.com/llvm/llvm-project 1a93ff55bc809487cd9f39ab6285df0a3019c8fc), ClangBuiltLinux LLD 17.0.0) #1 SMP PREEMPT Mon Jun 26 14:27:07 MST 2023
...
[    1.557056] Run /init as init process
[    1.575643] ------------[ cut here ]------------
[    1.579839] kernel BUG at arch/loongarch/kernel/traps.c:851!
[    1.580207] Oops - BUG[#1]:
[    1.580451] CPU: 0 PID: 1 Comm: init Not tainted 6.4.0-rc7-next-20230626+ #1
[    1.580738] Hardware name: QEMU QEMU Virtual Machine, BIOS unknown 2/2/2022
[    1.580962] pc 9000000000223e24 ra 9000000000223d90 tp 9000000100098000 sp 900000010009be90
[    1.581252] a0 0000000000000000 a1 0000000000402000 a2 0000000000400000 a3 0000000000000003
[    1.581489] a4 0000000000000000 a5 00007ffffc3901b8 a6 0000000000000004 a7 0000000000000030
[    1.581699] t0 90000000024d4850 t1 ffffffffffffffff t2 2424242424242424 t3 00007ffff3e93f48
[    1.581877] t4 7efefefefefefeff t5 810f070303010103 t6 4d4824160a4b570a t7 fffffffffefef000
[    1.582105] t8 ffffffffffffffff u0 900000000049cb94 s9 0000000000000000 s0 0000000000000000
[    1.582310] s1 00007ffff3ead2d0 s2 00007ffffbf8d8f0 s3 0000000000000000 s4 00007ffffbf8d8f0
[    1.582527] s5 0000555557d1febf s6 0000000000000000 s7 00007ffffbf8d8f0 s8 0000555557dfbdf0
[    1.582755]    ra: 9000000000223d90 init_restore_fp+0x50/0xe8
[    1.583280]   ERA: 9000000000223e24 init_restore_fp+0xe4/0xe8
[    1.583437]  CRMD: 000000b0 (PLV0 -IE -DA +PG DACF=CC DACM=CC -WE)
[    1.584064]  PRMD: 00000004 (PPLV0 +PIE -PWE)
[    1.584213]  EUEN: 00000001 (+FPE -SXE -ASXE -BTE)
[    1.584385]  ECFG: 00071c1c (LIE=2-4,10-12 VS=7)
[    1.584685] ESTAT: 000c0000 [BRK] (IS= ECode=12 EsubCode=0)
[    1.584871]  PRID: 0014c010 (Loongson-64bit, Loongson-3A5000)
[    1.585086] Process init (pid: 1, threadinfo=(____ptrval____), task=(____ptrval____))
[    1.585426] Stack : 900000010009bec0 90000000024d48a0 0000000000000000 00007ffff3ead228
[    1.585726]         00007ffffbf8de30 90000000033a1f24 0000000000000000 00007ffff3e93fc0
[    1.585927]         0000000000000000 00007ffffbf8d770 00007ffffbf8d7b0 0000000000000000
[    1.586147]         00007ffffbf8d998 0000000000000000 0000000000000000 00007ffffc3901b8
[    1.586340]         0000000000000004 0000000000000030 00007ffffbf8d7a0 00007ffff3e82b9c
[    1.586549]         2424242424242424 00007ffff3e93f48 7efefefefefefeff 810f070303010103
[    1.586761]         4d4824160a4b570a fffffffffefef000 ffffffffffffffff 0000000000000000
[    1.586978]         00007ffffbf8de30 00007ffff3ead228 00007ffff3ead2d0 00007ffffbf8d8f0
[    1.587198]         0000000000000000 00007ffffbf8d8f0 0000555557d1febf 0000000000000000
[    1.587406]         00007ffffbf8d8f0 0000555557dfbdf0 ffffffffffffff9c 00007ffff3e9ace4
[    1.587628]         ...
[    1.587727] Call Trace:
[    1.587831] [<9000000000223e24>] init_restore_fp+0xe4/0xe8
[    1.588206] [<90000000024d48a0>] do_fpu+0x50/0xa0
[    1.588353] [<90000000033a1f24>] exception_handlers+0x1f24/0x10000
[    1.588533] [<00007ffff3e9ace4>] 0x7ffff3e9ace4
[    1.588703]
[    1.588767] Code: 28c02061  02c04063  4c000020 <002a0001> 02ffc063  29c02061  29c00076  28c02044  14000405
[    1.589106]
[    1.590614] ---[ end trace 0000000000000000 ]---
[    1.590907] note: init[1] exited with preempt_count 1
[    1.591363] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
[    1.592013] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]---

$ git -C $CBL_SRC/linux-next log --oneline origin/master^..
4e3dc0ab3eab LoongArch: Use CLANG_FLAGS in vDSO instead of filtering out '--target='
226fd644a791 Makefile: Add loongarch target flag for Clang compilation
1a3f5dc5d49b LoongArch: Mark Clang LTO as working
e7e2acd4388f LoongArch: Tweak CFLAGS for Clang compatibility
72d195ec176e LoongArch: Simplify the invtlb wrappers
31be70e71c7d LoongArch: Make the CPUCFG and CSR ops simple aliases of compiler built-ins
b96f1a7e225a LoongArch: Prepare for assemblers with proper FCSR class support
d44d7d406ac0 LoongArch: extable: Also recognize ABI names of registers
32c459db8e9b LoongArch: Calculate various sizes in the linker script
60e7c4a25da6 Add linux-next specific files for 20230626

$ git -C $CBL_SRC/llvm-project log --oneline origin/main^..
1a93ff55bc80 D138135
7827beeebb59 [flang][hlfir] `hlfir.char_extremum` lowering

A kernel built using GCC 13.1.0 from kernel.org has no issues.

$ ./boot-qemu.py -k $TMP_BUILD_FOLDER/linux-next-gcc
...
[    1.572243] Run /init as init process
Starting syslogd: OK
Starting klogd: OK
Running sysctl: OK
Saving 256 bits of non-creditable seed for next boot
Starting network: OK
Linux version 6.4.0-rc7-next-20230626+ (nathan@dev-arch.thelio-3990X) (loongarch64-linux-gcc (GCC) 13.1.0, GNU ld (GNU Binutils) 2.40) #1 SMP PREEMPT Mon Jun 26 16:07:35 MST 2023
Stopping network: OK
Seeding 256 bits without crediting
Saving 256 bits of non-creditable seed for next boot
Stopping klogd: OK
Stopping syslogd: OK
umount: devtmpfs busy - remounted read-only
umount: can't unmount /: Invalid argument
The system is going down NOW!
Sent SIGTERM to all processes
Sent SIGKILL to all processes
Requesting system reboot
[    4.888685] ACPI: PM: Preparing to enter system sleep state S5
[    4.889428] reboot: Restarting system

Is this expected? I don't think I am missing a patch to LLVM, but I might be.

[xen0n]

Thanks for the quick follow-up. @nathanchance : Would you please share the config used to build the kernel? @chenhuacai told me yesterday privately that he backed out the KASan patches as they apparently need some more polish, but that happened after next-20230626, so I'm not sure if that's why but I'll try reproducing the crash locally.

[xen0n]

Thanks for the quick follow-up. @nathanchance : Would you please share the config used to build the kernel? @chenhuacai told me yesterday privately that he backed out the KASan patches as they apparently need some more polish, but that happened after next-20230626, so I'm not sure if that's why but I'll try reproducing the crash locally.

Oh I saw in the https://github.com/ClangBuiltLinux/linux/issues/1787 issue it should be the defconfig. I'll try with a setup similar to yours shortly (previously I did all Clang builds natively).

[nathanchance]

@xen0n If you have any issues reproducing, please let me know!

[xen0n]

I've reproduced and debugged the issue: it's https://github.com/llvm/llvm-project/issues/63549 and will get fixed by https://reviews.llvm.org/D153865.

[xen0n]

@nathanchance: FYI, next-20230628 now works with LLVM main branch (I tested https://github.com/llvm/llvm-project/commit/193d3ace2bdab6065959828d6b5fd9558a687eeb) + https://reviews.llvm.org/D138135 applied.

Regarding the QEMU options, it could be fine to continue running with -m 2G -smp 2 (i.e. the justification you put in the commit message is fair) but minimally just -m 1G would suffice and take slightly less resources. (The LoongArch virt machine needs at least 1G due to memory layout requirements.)

And thanks for your testing efforts!

[xen0n]

Ah there's already #109. I re-posted the feedback there for better visibility.