search

ucb-bar / chipyard

An Agile RISC-V SoC Design Framework with in-order cores, out-of-order cores, accelerators, and more
https://chipyard.readthedocs.io/en/stable/
BSD 3-Clause "New" or "Revised" License
1.57k stars 619 forks source link

How to generate fpga linux image for nvdla+rocket #1558

Closed Yuxin-Yu closed 1 year ago

Yuxin-Yu commented 1 year ago

Background Work

Chipyard Version and Hash

Release:1.10

OS Setup

In order to verify the functionality of nvdla in FPGA, I generated a rocket soc bit file which equipped with small nvdla. However, I am not sure how to generate the corresponding Linux image. According to the readme in nvdla workload. I switched the Linux repository in Firemarshal to the nvdla-linux-v57 branch and ran the instructions./marshall - v - d build ../nvdla-workload/marshal-config/nvdla-base. json,but the following error occurred:

(firemarshal) yyx@yyx-virtual-machine:~/riscv/projects/chipyard/software/firemarshal$ ./marshal -v build ../nvdla-workload/marshal-configs/nvdla-base.json 
DEBUG: Loading nvdla-base.json:/home/yyx/riscv/projects/chipyard/software/nvdla-workload/marshal-configs/nvdla-base.json
DEBUG: Failed checking if running in CYGWIN due to: FileNotFoundError(2, 'No such file or directory')
DEBUG: Popen(['git', 'cat-file', '--batch-check'], cwd=/home/yyx/riscv/projects/chipyard/software/firemarshal/boards/prototype/distros/br/buildroot, universal_newlines=False, shell=None, istream=<valid stream>)
DEBUG: Popen(['git', 'diff', '--cached', '--abbrev=40', '--full-index', '--raw'], cwd=/home/yyx/riscv/projects/chipyard/software/firemarshal/boards/prototype/distros/br/buildroot, universal_newlines=False, shell=None, istream=None)
DEBUG: Popen(['git', 'diff', '--abbrev=40', '--full-index', '--raw'], cwd=/home/yyx/riscv/projects/chipyard/software/firemarshal/boards/prototype/distros/br/buildroot, universal_newlines=False, shell=None, istream=None)
DEBUG: Popen(['git', 'diff', '--cached', '--abbrev=40', '--full-index', '--raw'], cwd=/home/yyx/riscv/projects/chipyard/software/firemarshal/boards/prototype/distros/br/buildroot, universal_newlines=False, shell=None, istream=None)
DEBUG: Popen(['git', 'diff', '--abbrev=40', '--full-index', '--raw'], cwd=/home/yyx/riscv/projects/chipyard/software/firemarshal/boards/prototype/distros/br/buildroot, universal_newlines=False, shell=None, istream=None)
DEBUG: Creating distro br : 222b
DEBUG: Popen(['git', 'cat-file', '--batch-check'], cwd=/home/yyx/riscv/projects/chipyard/software/firemarshal/boards/prototype/distros/br/buildroot, universal_newlines=False, shell=None, istream=<valid stream>)
DEBUG: Popen(['git', 'diff', '--cached', '--abbrev=40', '--full-index', '--raw'], cwd=/home/yyx/riscv/projects/chipyard/software/firemarshal/boards/prototype/distros/br/buildroot, universal_newlines=False, shell=None, istream=None)
DEBUG: Popen(['git', 'diff', '--abbrev=40', '--full-index', '--raw'], cwd=/home/yyx/riscv/projects/chipyard/software/firemarshal/boards/prototype/distros/br/buildroot, universal_newlines=False, shell=None, istream=None)
DEBUG: Popen(['git', 'diff', '--cached', '--abbrev=40', '--full-index', '--raw'], cwd=/home/yyx/riscv/projects/chipyard/software/firemarshal/boards/prototype/distros/br/buildroot, universal_newlines=False, shell=None, istream=None)
DEBUG: Popen(['git', 'diff', '--abbrev=40', '--full-index', '--raw'], cwd=/home/yyx/riscv/projects/chipyard/software/firemarshal/boards/prototype/distros/br/buildroot, universal_newlines=False, shell=None, istream=None)
Traceback (most recent call last):
  File "/home/yyx/riscv/projects/chipyard/software/firemarshal/wlutil/config.py", line 700, in __init__
    self._initializeFromBase(self.cfgs[cfgName])
  File "/home/yyx/riscv/projects/chipyard/software/firemarshal/wlutil/config.py", line 801, in _initializeFromBase
    cfg.applyBase(baseCfg)
  File "/home/yyx/riscv/projects/chipyard/software/firemarshal/wlutil/config.py", line 553, in applyBase
    inheritLinuxOpts(self.cfg, baseCfg)
  File "/home/yyx/riscv/projects/chipyard/software/firemarshal/wlutil/config.py", line 351, in inheritLinuxOpts
    for name, src in list(config['linux']['modules'].items()):
KeyError: 'modules'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/yyx/riscv/projects/chipyard/software/firemarshal/./marshal", line 233, in <module>
    main()
  File "/home/yyx/riscv/projects/chipyard/software/firemarshal/./marshal", line 117, in main
    cfgs = wlutil.ConfigManager(args.config_files, list(workdirs.keys()))
  File "/home/yyx/riscv/projects/chipyard/software/firemarshal/wlutil/config.py", line 702, in __init__
    raise WorkloadConfigError(cfgName,
wlutil.config.WorkloadConfigError: Unable to load workload nvdla-base.json: Missing required option 'modules'

Additionally, it should be noted that my marshal-config. yaml uses the prototype mode, and the content of this file is as follows: board-dir : 'boards/prototype' Secondly, it is normal for me to generate the default prototype configuration's liunx image in the firemarshal conda environment

Other Setup

No response

Current Behavior

no

Expected Behavior

no

Other Information

No response

Yuxin-Yu commented 1 year ago

Hi @abejgonzalez .I used a method to generate NVDLA's Linux image files . The specific process is as follows:

  1. Switch firemarshal's liunx image branch to nvdla-linux-v57
  2. Add the Linux information from the nvdla-workload/nvdla-base/nvdla-kfrag file to firemarshal/boards/prototype/base-workloads/br-base/Linux-config
  3. Keep marshal-config. yaml in prototype mode unchanged
  4. Generate corresponding Linux images using build and install in sequence However, during the process of starting Linux with the above image, I encountered the following issues: 
    
OpenSBI v1.2
____                    _____ ____ _____
/ __ \                  / ____|  _ \_   _|
| |  | |_ __   ___ _ __ | (___ | |_) || |
| |  | | '_ \ / _ \ '_ \ \___ \|  _ < | |
| |__| | |_) |  __/ | | |____) | |_) || |_
\____/| .__/ \___|_| |_|_____/|___/_____|
    | |
    |_|

Platform Name : freechips,rocketchip-unknown Platform Features : medeleg Platform HART Count : 1 Platform IPI Device : aclint-mswi Platform Timer Device : aclint-mtimer @ 1000000Hz Platform Console Device : sifive_uart Platform HSM Device : --- Platform PMU Device : --- Platform Reboot Device : --- Platform Shutdown Device : --- Platform Suspend Device : --- Firmware Base : 0x80000000 Firmware Size : 232 KB Firmware RW Offset : 0x20000 Runtime SBI Version : 1.0

Domain0 Name : root Domain0 Boot HART : 0 Domain0 HARTs : 0* Domain0 Region00 : 0x0000000002000000-0x000000000200ffff M: (I,R,W) S/U: () Domain0 Region01 : 0x0000000080000000-0x000000008001ffff M: (R,X) S/U: () Domain0 Region02 : 0x0000000080020000-0x000000008003ffff M: (R,W) S/U: () Domain0 Region03 : 0x0000000000000000-0xffffffffffffffff M: (R,W,X) S/U: (R,W,X) Domain0 Next Address : 0x0000000080200000 Domain0 Next Arg1 : 0x0000000081500000 Domain0 Next Mode : S-mode Domain0 SysReset : yes Domain0 SysSuspend : yes

Boot HART ID : 0 Boot HART Domain : root Boot HART Priv Version : v1.11 Boot HART Base ISA : rv64imafdcx Boot HART ISA Extensions : none Boot HART PMP Count : 8 Boot HART PMP Granularity : 4 Boot HART PMP Address Bits: 31 Boot HART MHPM Count : 2 Boot HART MIDELEG : 0x0000000000000222 Boot HART MEDELEG : 0x000000000000b109 [ 0.000000] OF: fdt: No chosen node found, continuing without [ 0.000000] OF: fdt: Ignoring memory range 0x80000000 - 0x80200000 [ 0.000000] Forcing kernel command line to: console=ttyS0 console=ttySIF0 earlycon [ 0.000000] Linux version 5.7.0-rc3-58540-g66e8cf3c569c (yyx@yyx-virtual-machine) (gcc version 12.2.0 (g2ee5e430018), GNU ld (GNU Binutils) 2.39) #7 SMP Tue Jul 18 09:07:51 CST 2023 [ 0.000000] Malformed early option 'earlycon' [ 0.000000] initrd not found or empty - disabling initrd [ 0.000000] Zone ranges: [ 0.000000] DMA32 [mem 0x0000000080200000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x000000017fffffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000080200000-0x000000017fffffff] [ 0.000000] Initmem setup node 0 [mem 0x0000000080200000-0x000000017fffffff] [ 0.000000] On node 0 totalpages: 1048064 [ 0.000000] DMA32 zone: 7161 pages used for memmap [ 0.000000] DMA32 zone: 0 pages reserved [ 0.000000] DMA32 zone: 523776 pages, LIFO batch:63 [ 0.000000] Normal zone: 7168 pages used for memmap [ 0.000000] Normal zone: 524288 pages, LIFO batch:63 [ 0.000000] software IO TLB: mapped [mem 0xfbfff000-0xfffff000] (64MB) [ 0.000000] SBI specification v1.0 detected [ 0.000000] SBI implementation ID=0x1 Version=0x10002 [ 0.000000] SBI v0.2 TIME extension detected [ 0.000000] SBI v0.2 IPI extension detected [ 0.000000] SBI v0.2 RFENCE extension detected [ 0.000000] SBI v0.2 HSM extension detected [ 0.000000] elf_hwcap is 0x112d [ 0.000000] percpu: Embedded 17 pages/cpu s29720 r8192 d31720 u69632 [ 0.000000] pcpu-alloc: s29720 r8192 d31720 u69632 alloc=17*4096 [ 0.000000] pcpu-alloc: [0] 0 [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1033735 [ 0.000000] Kernel command line: console=ttyS0 console=ttySIF0 earlycon [ 0.000000] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes, linear) [ 0.000000] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes, linear) [ 0.000000] Sorting ex_table... [ 0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off [ 0.000000] Memory: 4043800K/4192256K available (3034K kernel code, 3350K rwdata, 2048K rodata, 10077K init, 255K bss, 148456K reserved, 0K cma-reserved) [ 0.000000] Virtual kernel memory layout: [ 0.000000] fixmap : 0xffffffcefee00000 - 0xffffffceff000000 (2048 kB) [ 0.000000] pci io : 0xffffffceff000000 - 0xffffffcf00000000 ( 16 MB) [ 0.000000] vmemmap : 0xffffffcf00000000 - 0xffffffcfffffffff (4095 MB) [ 0.000000] vmalloc : 0xffffffd000000000 - 0xffffffdfffffffff (65535 MB) [ 0.000000] lowmem : 0xffffffe000000000 - 0xffffffe0ffe00000 (4094 MB) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=32 to nr_cpu_ids=1. [ 0.000000] rcu: RCU debug extended QS entry/exit. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1 [ 0.000000] NR_IRQS: 0, nr_irqs: 0, preallocated irqs: 0 [ 0.000000] plic: mapped 3 interrupts with 1 handlers for 2 contexts. [ 0.000000] riscv_timer_init_dt: Registering clocksource cpuid [0] hartid [0] [ 0.000000] clocksource: riscv_clocksource: mask: 0xffffffffffffffff max_cycles: 0x1d854df40, max_idle_ns: 3526361616960 ns [ 0.000127] sched_clock: 64 bits at 1000kHz, resolution 1000ns, wraps every 2199023255500ns [ 0.001983] Console: colour dummy device 80x25 [ 0.002640] Calibrating delay loop (skipped), value calculated using timer frequency.. 2.00 BogoMIPS (lpj=4000) [ 0.002949] pid_max: default: 32768 minimum: 301 [ 0.007338] Mount-cache hash table entries: 8192 (order: 4, 65536 bytes, linear) [ 0.008694] Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes, linear) [ 0.039806] rcu: Hierarchical SRCU implementation. [ 0.048598] smp: Bringing up secondary CPUs ... [ 0.048828] smp: Brought up 1 node, 1 CPU [ 0.057160] devtmpfs: initialized [ 0.078287] random: get_random_u32 called from bucket_table_alloc.isra.0+0x4a/0xbc with crng_init=0 [ 0.080008] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns [ 0.080456] futex hash table entries: 256 (order: 2, 16384 bytes, linear) [ 0.252079] clocksource: Switched to clocksource riscv_clocksource [ 2.013675] workingset: timestamp_bits=62 max_order=20 bucket_order=0 [ 2.188566] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253) [ 2.188797] io scheduler mq-deadline registered [ 2.188976] io scheduler kyber registered [ 2.189641] start plist test [ 2.330326] end plist test [ 3.236930] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled [ 3.257038] 64000000.serial: ttySIF0 at MMIO 0x64000000 (irq = 2, base_baud = 0) is a SiFive UART v0 [ 3.257279] BUG: spinlock bad magic on CPU#0, swapper/0/1 [ 3.257537] lock: 0xffffffe0fbdb5018, .magic: 00000000, .owner: /-1, .owner_cpu: 0 [ 3.257781] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.7.0-rc3-58540-g66e8cf3c569c #7 [ 3.257940] Call Trace: [ 3.258219] [] walk_stackframe+0x0/0xaa [ 3.258449] [] show_stack+0x2a/0x34 [ 3.258678] [] dump_stack+0x6c/0x86 [ 3.258944] [] spin_dump+0x66/0x72 [ 3.259169] [] do_raw_spin_lock+0xa8/0xc4 [ 3.259456] [] _raw_spin_lock_irqsave+0x20/0x2c [ 3.259697] [] uart_configure_port.isra.0+0x12a/0x1f4 [ 3.259915] [] uart_add_one_port+0x152/0x202 [ 3.260188] [] sifive_serial_probe+0x174/0x244 [ 3.260464] [] platform_drv_probe+0x2e/0x5a [ 3.260687] [] really_probe+0x9c/0x260 [ 3.260916] [] driver_probe_device+0x6e/0x84 [ 3.261144] [] device_driver_attach+0x7e/0x82 [ 3.261366] [] driver_attach+0x3a/0x96 [ 3.261579] [] bus_for_each_dev+0x40/0x64 [ 3.261796] [] driver_attach+0x1a/0x22 [ 3.262015] [] bus_add_driver+0xce/0x180 [ 3.262240] [] driver_register+0x48/0xd8 [ 3.262478] [] __platform_driver_register+0x3a/0x42 [ 3.262721] [] sifive_serial_init+0x30/0x50 [ 3.263010] [] do_one_initcall+0x2e/0x134 [ 3.263237] [] do_initcalls+0xea/0x11a [ 3.263461] [] kernel_init_freeable+0x8e/0xd6 [ 3.263684] [] kernel_init+0x12/0x118 [ 3.263944] [] ret_from_exception+0x0/0xc [ 3.884273] printk: console [ttySIF0] enabled [ 4.078182] loop: module loaded [ 4.084841] sifive_spi 64001000.spi: mapped; irq=3, cs=1 [ 4.093382] sifive_spi 64001000.spi: registered master spi0 [ 4.101599] spi spi0.0: setup mode 0, 8 bits/w, 1000000 Hz max --> 0 [ 4.110577] sifive_spi 64001000.spi: registered child spi0.0 [ 4.379908] Freeing unused kernel memory: 10076K [ 4.387726] Run /init as init process [ 4.391177] with arguments: [ 4.393761] /init [ 4.396351] with environment: [ 4.399511] HOME=/ [ 4.401523] TERM=linux Running FireMarshal nodisk init Loading FireMarshal platform drivers Calling distro init Starting syslogd: OK Starting klogd: OK Running sysctl: OK Starting mdev... OK Saving random seed: [ 13.107270] random: dd: uninitialized urandom read (512 bytes read) OK Starting network: ip: socket: Function not implemented ip: socket: Function not implemented FAIL launching firemarshal workload run/command firemarshal workload run/command done

Welcome to Buildroot buildroot login: root Password: Login incorrect login[100]: invalid password for 'root' on 'console' buildroot login: buildroot login: root Password: Login incorrect login[100]: invalid password for 'root' on 'console'


During the Linux startup process, there was a kernel error, and I was unable to log in using root+fpga (theoretical account and password) or root+firesim on the login console.
Additionally, based on the description in question https://github.com/ucb-bar/chipyard/issues/573, I have attempted the following two methods:
1. Add the following Linux configuration information to firemarshal/boards/prototype/base workloads/br base/Linux configuration:

CONFIG_DEBUG_RT_MUTEXES=y CONFIG_DEBUG_SPINLOCK=y CONFIG_DEBUG_MUTEXES=y CONFIG_DEBUG_RWSEMS=y CONFIG_DEBUG_ATOMIC_SLEEP=y


2. Try switching the Rocket kernel to the Boom kernel.
Unfortunately, after trying the above two methods, the kernel still reported the same error and was unable to access Linux
Yuxin-Yu commented 1 year ago

I have resolved the above issue and made the following modifications:

  1. Based on https://github.com/ucb-bar/nvdla-workload/issues/2, I have changed the debug related configuration information to the following: 
    CONFIG_DEBUG_RT_MUTEXES=n
CONFIG_DEBUG_SPINLOCK=n
CONFIG_DEBUG_MUTEXES=n
CONFIG_DEBUG_RWSEMS=n
CONFIG_DEBUG_ATOMIC_SLEEP=n
  2. As my bit file was generated using version 1.8.1, I switched to firemarshal under the 1.8.1 chipyard project. Finally, I successfully started Linux and installed the opendla. ko driver image

And I am not sure if the issue with login failure on Linux is due to a mismatch between the bit file and the Linux image, or if there is a problem with the latest version of Firemarshal