Closed paranlee closed 11 months ago
namhyung
commented
2 years ago One concern is maintenance burden since I'm not familiar with the architecture details. If you want to work on it, I hope you can support the arch issues (if any) for a long time.
paranlee
commented
1 year ago To Find arch specific gcc option -pg's mcount() compatables, Done with tests/s-abc.c builded with gcc -pg -O0 -g3 -o t-abc tests/s-abc.c.
$ uname -a
Linux ubuntu 6.2.0-19-generic #19.1-Ubuntu SMP Fri Mar 31 12:41:53 UTC 2023 riscv64 riscv64 riscv64 GNU/Linux
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 23.04"
NAME="Ubuntu"
VERSION_ID="23.04"
VERSION="23.04 (Lunar Lobster)"static int a(void)
{
8b0: 1141 add sp,sp,-16
8b2: e406 sd ra,8(sp)
8b4: e022 sd s0,0(sp)
8b6: 0800 add s0,sp,16
8b8: 8786 mv a5,ra
8ba: 853e mv a0,a5
8bc: ee5ff0ef jal 7a0 <_mcount@plt>
return b() - 1;
8c0: 014000ef jal 8d4 <b>
8c4: 87aa mv a5,a0
8c6: 37fd addw a5,a5,-1
8c8: 2781 sext.w a5,a5
}
8ca: 853e mv a0,a5
8cc: 60a2 ld ra,8(sp)
8ce: 6402 ld s0,0(sp)
8d0: 0141 add sp,sp,16
8d2: 8082 ret
00000000000008d4 <b>:
static int b(void)
{
8d4: 1141 add sp,sp,-16
8d6: e406 sd ra,8(sp)
8d8: e022 sd s0,0(sp)
8da: 0800 add s0,sp,16
8dc: 8786 mv a5,ra
8de: 853e mv a0,a5
8e0: ec1ff0ef jal 7a0 <_mcount@plt>
return c() + 1;
8e4: 014000ef jal 8f8 <c>
8e8: 87aa mv a5,a0
8ea: 2785 addw a5,a5,1
8ec: 2781 sext.w a5,a5
}
8ee: 853e mv a0,a5
8f0: 60a2 ld ra,8(sp)
8f2: 6402 ld s0,0(sp)
8f4: 0141 add sp,sp,16
8f6: 8082 ret$ uname -a
Linux paran-virtual-machine 6.3.0-7-generic #7-Ubuntu SMP PREEMPT_DYNAMIC Thu Jun 8 16:02:30 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/os-release
PRETTY_NAME="Ubuntu Mantic Minotaur (development branch)"
NAME="Ubuntu"
VERSION_ID="23.10"
VERSION="23.10 (Mantic Minotaur)"static int a(void)
{
1229: f3 0f 1e fa endbr64
122d: 55 push %rbp
122e: 48 89 e5 mov %rsp,%rbp
1231: ff 15 b1 2d 00 00 call *0x2db1(%rip) # 3fe8 <mcount@GLIBC_2.2.5>
return b() - 1;
1237: e8 05 00 00 00 call 1241 <b>
123c: 83 e8 01 sub $0x1,%eax
}
123f: 5d pop %rbp
1240: c3 ret
0000000000001241 <b>:
static int b(void)
{
1241: f3 0f 1e fa endbr64
1245: 55 push %rbp
1246: 48 89 e5 mov %rsp,%rbp
1249: ff 15 99 2d 00 00 call *0x2d99(%rip) # 3fe8 <mcount@GLIBC_2.2.5>
return c() + 1;
124f: e8 05 00 00 00 call 1259 <c>
1254: 83 c0 01 add $0x1,%eax
}
1257: 5d pop %rbp
1258: c3 ret$ uname -a
Linux raspberrypi 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr 3 17:24:16 BST 2023 aarch64 GNU/Linux
$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11static int a(void)
{
934: a9bf7bfd stp x29, x30, [sp, #-16]!
938: 910003fd mov x29, sp
93c: aa1e03e0 mov x0, x30
940: aa0003fe mov x30, x0
944: d50320ff xpaclri
948: aa1e03e0 mov x0, x30
94c: 97ffff99 bl 7b0 <_mcount@plt>
return b() - 1;
950: 94000004 bl 960 <b>
954: 51000400 sub w0, w0, #0x1
}
958: a8c17bfd ldp x29, x30, [sp], #16
95c: d65f03c0 ret
0000000000000960 <b>:
static int b(void)
{
960: a9bf7bfd stp x29, x30, [sp, #-16]!
964: 910003fd mov x29, sp
968: aa1e03e0 mov x0, x30
96c: aa0003fe mov x30, x0
970: d50320ff xpaclri
974: aa1e03e0 mov x0, x30
978: 97ffff8e bl 7b0 <_mcount@plt>
return c() + 1;
97c: 94000004 bl 98c <c>
980: 11000400 add w0, w0, #0x1
}
984: a8c17bfd ldp x29, x30, [sp], #16
988: d65f03c0 retThanks for the update, it'd be nice if you could link the psABI or calling convention document for RISC-V.
paranlee
commented
1 year ago @namhyung I have studied Linux Kernel & existing implementation of uftrace other arch for RISC-V implementation. Thank you!
paranlee
commented
1 year ago We can full virtualize the RISC-V 64bit Development Platform. :)
gichoel
commented
1 year ago If you are interested in documentation on building a Windows Based RISC-V development environment in korean, please see the links below.
honggyukim
commented
1 year ago Hi @gichoel, thanks very much for the useful document. But it looks the environment is based on Windows. It'd also be useful if there is a guide for Linux users.
In addition, it looks a bit complicated to extract some files such as fw_jump.elf, uboot.elf, etc. Is it possible you to upload those files then provide a link?
gichoel
commented
1 year ago Based on your feedback, I'll be adding documentation for the Linux version and a simple way to use files like fw_jump.elf, uboot.elf, etc.
gichoel
commented
1 year ago If you are interested in documentation on building a Ubuntu Linux Based RISC-V Cross-Compile development environment in korean, please see the links below.
honggyukim
commented
1 year ago Thanks very much for the document. I'm able to setup riscv64 environment following your guide.
But I think we can further simplify the steps as follows.
$ sudo apt install qemu-system-riscv64 qemu-utils$ sudo apt install opensbi u-boot-qemu
$ wget https://cdimage.ubuntu.com/releases/22.04/release/ubuntu-22.04.3-preinstalled-server-riscv64+unmatched.img.xz$ xz -d ubuntu-22.04.3-preinstalled-server-riscv64+unmatched.img.xz
$ qemu-img resize ubuntu-22.04.3-preinstalled-server-riscv64+unmatched.img 12G
...
Image resized.
4. Prepare a script to boot riscv64 ubuntu image in qemu$ echo "qemu-system-riscv64 -M virt -smp 2 -m 4096 -nographic -bios /usr/lib/riscv64-linux-gnu/opensbi/generic/fw_jump.elf -kernel /usr/lib/u-boot/qemu-riscv64_smode/uboot.elf -drive if=none,file=ubuntu-22.04.3-preinstalled-server-riscv64+unmatched.img,format=raw,id=hd0 -device virtio-blk-device,drive=hd0 -device virtio-net-device,netdev=net0 -netdev user,hostfwd=tcp:127.0.0.1:2222-:22,id=net0" > riscv64-ubuntu-22.04.sh
$ chmod +x riscv64-ubuntu-22.04.sh
5. Run the script to boot qemu riscv64 image$ ./riscv64-ubuntu-22.04.sh
OpenSBI v1.3
| / \ / __ | _ | _ __ | (___ | _) | ' \ / \ '_ \ ___ \ | _ < | __ | _) | __/ | ____) | _) | _ ____/ | ./ _ | _ | _ | _____/ | _/___ | ||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| _ |
Platform Name : riscv-virtio,qemu Platform Features : medeleg Platform HART Count : 2 Platform IPI Device : aclint-mswi Platform Timer Device : aclint-mtimer @ 10000000Hz Platform Console Device : semihosting ...
6. Change password after logging in as a user name ubuntu ...Ubuntu 22.04.3 LTS ubuntu ttyS0
ubuntu login: ubuntu Password: You are required to change your password immediately (administrator enforced). Changing password for ubuntu. Current password: New password: Retype new password:
7. Connect the riscv64 machine through ssh$ ssh -p 2222 ubuntu@localhost ubuntu@localhost's password: Welcome to Ubuntu 22.04.3 LTS (GNU/Linux 5.19.0-1021-generic riscv64) ... ubuntu@ubuntu:~$ uname -m riscv64
8. Download and prepare for building uftrace inside the riscv64 world.ubuntu@ubuntu:~$ sudo apt install build-essential
ubuntu@ubuntu:~$ git clone https://github.com/namhyung/uftrace && cd uftrace
ubuntu@ubuntu:~/uftrace$ ./configure uftrace detected system features: ... prefix: /usr/local ... libelf: [ OFF ] - more flexible ELF data handling ... libdw: [ OFF ] - DWARF debug info support ... libpython: [ OFF ] - python tracing & scripting support ... libluajit: [ OFF ] - luajit scripting support ... libncursesw: [ OFF ] - TUI support ... cxa_demangle: [ on ] - full demangler support with libstdc++ ... perf_event: [ on ] - perf (PMU) event support ... schedule: [ on ] - scheduler event support ... capstone: [ OFF ] - full dynamic tracing support ... libunwind: [ OFF ] - stacktrace support (optional for debugging)
ubuntu@ubuntu:~/uftrace$ cat .config | grep ARCH override ARCH := riscv64
9. Build uftrace and see the compilation error.ubuntu@ubuntu:~/uftrace$ make -j2
CC cmds/graph.o
CC cmds/live.o
CC cmds/record.o
CC cmds/recv.o
CC cmds/replay.o
CC cmds/report.o
CC cmds/script.o
CC cmds/tui.o
CC utils/argspec.o
CC utils/auto-args.o
CC utils/data-file.o
CC utils/debug.o
CC utils/demangle.o
CC utils/dwarf.o
/home/ubuntu/uftrace/utils/dwarf.c:15:10: fatal error: mcount-arch.h: No such file or directory
15 | #include "mcount-arch.h"
| ^~~~~~~
compilation terminated.
I think we better build and test inside riscv64 qemu. It might be a bit slower than host machine, but the development and test cycle is much better than cross compilation environment.
honggyukim
commented
1 year ago It's a different story but it might also be very helpful if we can have a qemu android development environment as well.
gichoel
commented
1 year ago Thanks for the great suggestion to further simplify the steps :)
It's a different story but it might also be very helpful if we can have a qemu android development environment as well.
It seems like a great idea from the perspective of having a build and test environment for multiple architectures and operating systems on one local Linux machine, since users most likely don't have all the development boards or machines.
I'll give it a try when I get some time.
MichelleJin12
commented
1 year ago qemu-img
sudo apt-get install qemu-utilsqemu-system-riscv64
sudo apt install qemu-system-miscThere is libluajit issue for adding support RISC-V. Reference for using libluajit in RISC-V in the future.
honggyukim
commented
1 year ago
- Install
qemu-imgsudo apt-get install qemu-utils
Thanks. I've updated the comment with qemu-utils.
paranlee
commented
1 year ago We can build latest QEMU, sometimes Linux Kernel & QEMU & boot firmware are version dependent :( Below Command Make QEMU IMG Utils & QEMU RISC-V runtime with network forwarding supports.
$ sudo apt -y install build-essential libglib2.0-dev libfdt-dev libpixman-1-dev zlib1g-dev libslirp-dev libcap-ng-dev libattr1-dev ninja-build
$ git clone https://github.com/qemu/qemu.git
$ mkdir build
$ cd build
$ ../configure --target-list=riscv64-softmmu --prefix="$HOME/MyUsrBin" --enable-slirp --enable-kvm --enable-virtfs --enable-tools
$ make -j $(nproc)
$ make installWe can find latest RISC-V google android info :)
paranlee
commented
1 year ago We can emulate RISC-V android 12 ;)
$ emulator -verbose -no-boot-anim -show-kernel -noaudio -selinux permissive -qemu -smp 1 -m 3584M -bios kernel/prebuilts/5.10/riscv64/fw_jump.binThe host distribution or operating system environment of the WSL doesn't really matter, but I thought it would be nice to match the redhat-like commands of the guest OS, so I done with fedora.
This guide has been rewritten with Fedora's RISC-V Boot_under_QEMU Guide as a reference.
Here is the last tested configuration for the instructions below:
Fedora 38
$ qemu-system-riscv64 --version
QEMU emulator version 7.2.6 (qemu-7.2.6-1.fc38)
Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developersWe will use Fedora-Developer-38-20230519.n.0.
Download disk image and verify
$ wget https://dl.fedoraproject.org/pub/alt/risc-v/disk_images/Fedora-Developer-38-20230519.n.0.SiFive.Unmatched.and.QEMU/Fedora-Developer-38-20230519.n.0-nvme.raw.img.xz
$ wget https://dl.fedoraproject.org/pub/alt/risc-v/disk_images/Fedora-Developer-38-20230519.n.0.SiFive.Unmatched.and.QEMU/Fedora-Developer-38-20230519.n.0-nvme.raw.img.xz.sha512sum
# Verify
sha512sum -c *.sha512sum
# Uncompress
unxz Fedora-Developer-38-20230519.n.0-nvme.raw.img.xzDownload the latest U-Boot (we need U-Boot SPL and U-Boot ITB files)
mkdir u
pushd u
rpm2cpio http://fedora.riscv.rocks/kojifiles/packages/uboot-tools/2023.01/2.4.riscv64.fc37/noarch/uboot-images-riscv64-2023.01-2.4.riscv64.fc37.noarch.rpm | cpio -dvim
popdUpdate the extlinux.conf with some tools(virt-edit or kpartx).
# We need to modify extlinux.conf for QEMU
$ virt-edit -a Fedora-Developer-38-20230519.n.0-nvme.raw.img /boot/extlinux/extlinux.confWe will use first solution virt-edit (We have not tested with kpartx).
# Or use kpartx:
mkdir -p /tmp/disk_img
sudo kpartx -a -v Fedora-Developer-38-20230519.n.0-nvme.raw.img
# Output example
add map loop2p1 (253:4): 0 1433600 linear 7:2 34
add map loop2p2 (253:5): 0 19537853 linear 7:2 1433634
# We need to mount the 1st partition (or /boot partition)
sudo mount /dev/mapper/loop2p1 /tmp/disk_img
# Edit extlinux.conf
sudo nvim /tmp/disk_img/extlinux/extlinux.confWe want to remove fdtdir because we will get DTB from the QEMU itself.
Set console to ttyS0.
So, we will remove fdtline.
# Existing boot entry
9 label Fedora Linux (6.4.12-200.0.riscv64.fc38.riscv64) 38 (Thirty Eight)
10 kernel /vmlinuz-6.4.12-200.0.riscv64.fc38.riscv64
11 append ro root=UUID=56f56257-a0d4-4865-a531-ade6042a110c rhgb LANG=en_US.UTF-8 console=ttySIF0,115200 earlycon nokaslr kmemleak=on
12 fdtdir /dtb-6.4.12-200.0.riscv64.fc38.riscv64/
13 initrd /6.4.12-200.0.riscv64.fc38.riscv64.img
# Modified
9 label Fedora Linux (6.4.12-200.0.riscv64.fc38.riscv64) 38 (Thirty Eight)
10 kernel /vmlinuz-6.4.12-200.0.riscv64.fc38.riscv64
11 append ro root=UUID=56f56257-a0d4-4865-a531-ade6042a110c rhgb LANG=en_US.UTF-8 console=ttySIF0,115200 earlycon nokaslr kmemleak=on
12 initrd /6.4.12-200.0.riscv64.fc38.riscv64.imgFinally, In our extlinux.conf. (Just removed fdtline)
# extlinux.conf generated by appliance-creator
ui menu.c32
menu autoboot Welcome to Fedora-Developer-38-20230519.n.0. Automatic boot in # second{,s}. Press a key for options.
menu title Fedora-Developer-38-20230519.n.0 Boot Options.
menu hidden
timeout 20
totaltimeout 600
default=Fedora-Developer-38-20230519.n.0 (6.2.16-300.0.riscv64.fc38.riscv64)
label Fedora Linux (6.4.12-200.0.riscv64.fc38.riscv64) 38 (Thirty Eight)
kernel /vmlinuz-6.4.12-200.0.riscv64.fc38.riscv64
append ro root=UUID=56f56257-a0d4-4865-a531-ade6042a110c rhgb LANG=en_US.UTF-8 console=ttySIF0,115200 earlycon nokaslr kmemleak=on
initrd /initramfs-6.4.12-200.0.riscv64.fc38.riscv64.imgUnmount
sudo sync
sudo umount /tmp/disk_img
sudo kpartx -d -v Fedora-Developer-37-20221130.n.0-nvme.raw.imgLaunch QEMU
Optional: Add QMP Option (QEMU Guest Dump Solution)
$ cat qemu-riscv64-fedora-38.sh
qemu-system-riscv64 \
-bios u/usr/share/uboot/qemu-riscv64_spl/u-boot-spl.bin \
-nographic \
-machine virt \
-smp 4 \
-m 4G \
-device loader,file=u/usr/share/uboot/qemu-riscv64_spl/u-boot.itb,addr=0x80200000 \
-object rng-random,filename=/dev/urandom,id=rng0 \
-device virtio-rng-device,rng=rng0 \
-device virtio-blk-device,drive=hd0 \
-drive file=Fedora-Developer-38-20230519.n.0-nvme.raw.img,format=raw,id=hd0 \
-device virtio-net-device,netdev=usernet \
-netdev user,id=usernet,hostfwd=tcp::10000-:22 \
-qmp tcp:localhost:4444,server,nowaitOnce machine is booted you can connect via SSH:
If you use it and are worried about it, be sure to change your ID and Password.
cat qemu-riscv64-fedora-38-ssh.sh
# ID: riscv
# Password: fedora_rocks!
ssh -p 10000 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o PreferredAuthentications=password -o PubkeyAuthentication=no riscv@localhostDevice Tree dump.
qemu-system-riscv64 -nographic -smp 8 -m 2G -machine virt,dumpdtb=qemu-riscv.dtb
dtc -I dtb -O dts qemu-riscv.dtb -o qemu-riscv.dtsThis can run up to 32 CPUs.
namhyung
commented
11 months ago We need dynamic tracing support but let's leave it in a separate issue (#1862).
I think add uftrace riscv support like other architectures already implemented is futuristic fantasy.
I am currently researching arm64 and riscv on the topic of architecture-dependent
mcountimplementation. and I found ftrace mcount implementation of the riscv architecture Linux kernel.Even if there is no riscv target board, full virtualization is possible in the QEMU environment, so the development environment does not seem to be a problem.
arch/riscv/kernel/mcount.Sarch/riscv/kernel/mcount-dyn.S