eugene-tarassov / vivado-risc-v

Xilinx Vivado block designs for FPGA RISC-V SoC running Debian Linux distro
858 stars 196 forks source link

Issues with NFS Boot on Xilinx Alveo U250 - SFP Port Not Lighting Up & DHCP #257

Open Xu-feng-feng opened 1 day ago

Xu-feng-feng commented 1 day ago

Hi, Eugene Thank you for the open-source project. Could anyone provide insight into the possible causes of this issue or suggest a solution? I have encountered an issue while attempting to boot a system on a Xilinx Alveo U250 using NFS. After generating the bitstream and downloading it to the XCU250, I noticed that the SFP port connected to the router does not light up.

I followed the instructions in [this issue thread](https://github.com/eugene-tarassov/vivado-risc-v/issues/62), and ran the following command:

make BOARD=u250 CONFIG= rocket64b2 \
ROOTFS_URL=192.168.1.100:/home/nfsroot \
HW_SERVER_ADDR=localhost:3121 JTAG_BOOT=1 jtag-boot

I’m using Vivado version 2023.2 for this project. However, the process stops at:

loop: module loaded

The full boot log looks as follows:

RISC-V 64, Boot ROM V3.8

OpenSBI v1.4
   ____                    _____ ____ _____
  / __ \                  / ____|  _ \_   _|
 | |  | |_ __   ___ _ __ | (___ | |_) || |
 | |  | | '_ \ / _ \ '_ \ \___ \|  _ < | |
 | |__| | |_) |  __/ | | |____) | |_) || |_
  \____/| .__/ \___|_| |_|_____/|____/_____|
        | |
        |_|

Platform Name             : freechips,rocketchip-vivado
Platform Features         : medeleg
Platform HART Count       : 2
Platform IPI Device       : aclint-mswi
Platform Timer Device     : aclint-mtimer @ 1250000Hz
Platform Console Device   : axi-uart
Platform HSM Device       : ---
Platform PMU Device       : ---
Platform Reboot Device    : ---
Platform Shutdown Device  : ---
Platform Suspend Device   : ---
Platform CPPC Device      : ---
Firmware Base             : 0x80000000
Firmware Size             : 192 KB
Firmware RW Offset        : 0x20000
Firmware RW Size          : 64 KB
Firmware Heap Offset      : 0x27000
Firmware Heap Size        : 36 KB (total), 2 KB (reserved), 9 KB (used), 24 KB (free)
Firmware Scratch Size     : 4096 B (total), 320 B (used), 3776 B (free)
Runtime SBI Version       : 2.0

Domain0 Name              : root
Domain0 Boot HART         : 0
Domain0 HARTs             : 0*,1*
Domain0 Region00          : 0x0000000002000000-0x000000000200ffff M: (I,R,W) S/U: ()
Domain0 Region01          : 0x0000000080020000-0x000000008002ffff M: (R,W) S/U: ()
Domain0 Region02          : 0x0000000080000000-0x000000008001ffff M: (R,X) S/U: ()
Domain0 Region03          : 0x000000000c000000-0x000000000fffffff M: (I,R,W) S/U: (R,W)
Domain0 Region04          : 0x0000000000000000-0xffffffffffffffff M: () S/U: (R,W,X)
Domain0 Next Address      : 0x0000000080200000
Domain0 Next Arg1         : 0x0000000000010080
Domain0 Next Mode         : S-mode
Domain0 SysReset          : yes
Domain0 SysSuspend        : yes

Boot HART ID              : 0
Boot HART Domain          : root
Boot HART Priv Version    : v1.12
Boot HART Base ISA        : rv64imafdcx
Boot HART ISA Extensions  : none
Boot HART PMP Count       : 8
Boot HART PMP Granularity : 2 bits
Boot HART PMP Address Bits: 32
Boot HART MHPM Info       : 0 (0x00000000)
Boot HART MIDELEG         : 0x0000000000000222
Boot HART MEDELEG         : 0x000000000000b109

U-Boot 2022.01-dirty (Nov 12 2024 - 15:40:49 +0800)

CPU:   rv64imafdczicsr_zifencei_zihpm_xrocket
Model: freechips,rocketchip-vivado
DRAM:  14 GiB
MMC:   
Loading Environment from nowhere... OK
In:    uart@60010000
Out:   uart@60010000
Err:   uart@60010000
Net:   No ethernet found.
Hit any key to stop autoboot:  0 
Moving Image from 0x81000000 to 0x80200000, end=8151f000
## Flattened Device Tree blob at 00010080
   Booting using the fdt blob at 0x010080
   Loading Device Tree to 0000000084ffc000, end 0000000084fffe37 ... OK

Starting kernel ...

[    0.000000] Linux version 6.9.12-dirty (find-fpga@findfpga-System-Product-Name) (riscv64-linux-gnu-gcc (Ubuntu 9.4.0-1ubuntu1~20.04) 9.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #1 SMP Tue Nov 12 12:51:18 CST 2024
[    0.000000] OF: fdt: Ignoring memory range 0x80000000 - 0x80200000
[    0.000000] Machine model: freechips,rocketchip-vivado
[    0.000000] SBI specification v2.0 detected
[    0.000000] SBI implementation ID=0x1 Version=0x10004
[    0.000000] SBI TIME extension detected
[    0.000000] SBI IPI extension detected
[    0.000000] SBI RFENCE extension detected
[    0.000000] SBI DBCN extension detected
[    0.000000] earlycon: axi_uart0 at MMIO 0x0000000060010000 (options '')
[    0.000000] printk: legacy bootconsole [axi_uart0] enabled
[    0.000000] efi: UEFI not found.
[    0.000000] Zone ranges:
[    0.000000]   DMA32    [mem 0x0000000080200000-0x00000000ffffffff]
[    0.000000]   Normal   [mem 0x0000000100000000-0x00000003ffffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080200000-0x00000003ffffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080200000-0x00000003ffffffff]
[    0.000000] On node 0, zone DMA32: 512 pages in unavailable ranges
[    0.000000] SBI HSM extension detected
[    0.000000] Falling back to deprecated "riscv,isa"
[    0.000000] riscv: base ISA extensions acdfim
[    0.000000] riscv: ELF capabilities acdfim
[    0.000000] percpu: Embedded 23 pages/cpu s56296 r8192 d29720 u94208
[    0.000000] Kernel command line: root=/dev/nfs rootfstype=nfs rw nfsroot=10.23.13.173:/sda2/nfsroot,nolock,vers=4,tcp ip=dhcp earlycon console=ttyAU0,115200n8 locale.LANG=en_US.UTF-8
[    0.000000] Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes, linear)
[    0.000000] Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes, linear)
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 3619335
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] software IO TLB: area num 2.
[    0.000000] software IO TLB: mapped [mem 0x00000000fbfff000-0x00000000fffff000] (64MB)
[    0.000000] Memory: 14347232K/14678016K available (6896K kernel code, 4792K rwdata, 4096K rodata, 2181K init, 315K bss, 330784K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-1, MinObjects=0, CPUs=2, Nodes=1
[    0.000000] rcu: Hierarchical RCU implementation.
[    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=32 to nr_cpu_ids=2.
[    0.000000]  Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
[    0.000000] RCU Tasks Trace: Setting shift to 1 and lim to 1 rcu_task_cb_adjust=1.
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] riscv-intc: 64 local interrupts mapped
[    0.000000] riscv: providing IPIs using SBI IPI extension
[    0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[    0.000000] clocksource: riscv_clocksource: mask: 0xffffffffffffffff max_cycles: 0x127350b88, max_idle_ns: 1763180808480 ns
[    0.000017] sched_clock: 64 bits at 1250kHz, resolution 800ns, wraps every 4398046510800ns
[    0.009816] Calibrating delay loop (skipped), value calculated using timer frequency.. 2.50 BogoMIPS (lpj=12500)
[    0.020449] pid_max: default: 32768 minimum: 301
[    0.028869] Mount-cache hash table entries: 32768 (order: 6, 262144 bytes, linear)
[    0.039406] Mountpoint-cache hash table entries: 32768 (order: 6, 262144 bytes, linear)
[    0.073608] riscv: ELF compat mode unsupported
[    0.073753] ASID allocator disabled (0 bits)
[    0.084916] rcu: Hierarchical SRCU implementation.
[    0.090076] rcu:     Max phase no-delay instances is 1000.
[    0.103345] EFI services will not be available.
[    0.111276] smp: Bringing up secondary CPUs ...
[    0.127271] smp: Brought up 1 node, 2 CPUs
[    0.152416] devtmpfs: initialized
[    0.178453] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.188808] futex hash table entries: 512 (order: 3, 32768 bytes, linear)
[    0.207892] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[    0.220674] DMA: preallocated 2048 KiB GFP_KERNEL pool for atomic allocations
[    0.229778] DMA: preallocated 2048 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    0.242528] thermal_sys: Registered thermal governor 'step_wise'
[    0.242649] thermal_sys: Registered thermal governor 'user_space'
[    0.319912] cpu1: Ratio of byte access time to unaligned word access is 0.01, unaligned accesses are slow
[    0.401364] cpu0: Ratio of byte access time to unaligned word access is 0.01, unaligned accesses are slow
[    0.457060] SCSI subsystem initialized
[    0.467240] usbcore: registered new interface driver usbfs
[    0.473799] usbcore: registered new interface driver hub
[    0.479736] usbcore: registered new device driver usb
[    0.498842] vgaarb: loaded
[    0.504892] clocksource: Switched to clocksource riscv_clocksource
[    0.541788] NET: Registered PF_INET protocol family
[    0.570192] IP idents hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    0.918332] tcp_listen_portaddr_hash hash table entries: 8192 (order: 5, 131072 bytes, linear)
[    0.929536] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)
[    0.938012] TCP established hash table entries: 131072 (order: 8, 1048576 bytes, linear)
[    0.967819] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, linear)
[    1.009785] TCP: Hash tables configured (established 131072 bind 65536)
[    1.018796] UDP hash table entries: 8192 (order: 6, 262144 bytes, linear)
[    1.030712] UDP-Lite hash table entries: 8192 (order: 6, 262144 bytes, linear)
[    1.044261] NET: Registered PF_UNIX/PF_LOCAL protocol family
[    1.057185] RPC: Registered named UNIX socket transport module.
[    1.063219] RPC: Registered udp transport module.
[    1.068433] RPC: Registered tcp transport module.
[    1.073212] RPC: Registered tcp-with-tls transport module.
[    1.079008] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    1.085806] PCI: CLS 0 bytes, default 64
[    1.102674] workingset: timestamp_bits=62 max_order=22 bucket_order=0
[    1.112576] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    1.123025] NFS: Registering the id_resolver key type
[    1.129287] Key type id_resolver registered
[    1.133563] Key type id_legacy registered
[    1.138274] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[    1.145476] nfs4flexfilelayout_init: NFSv4 Flexfile Layout Driver Registering...
[    1.156132] 9p: Installing v9fs 9p2000 file system support
[    1.164141] NET: Registered PF_ALG protocol family
[    1.170462] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
[    1.178566] io scheduler mq-deadline registered
[    1.183205] io scheduler kyber registered
[    1.193132] riscv-plic c000000.interrupt-controller: mapped 8 interrupts with 2 handlers for 4 contexts.
[    2.446349] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[    2.481648] 60010000.uart: ttyAU0 at MMIO 0x60010000 (irq = 12, base_baud = 0) is a axi-uart
[    2.490822] printk: legacy console [ttyAU0] enabled
[    2.490822] printk: legacy console [ttyAU0] enabled
[    2.501036] printk: legacy bootconsole [axi_uart0] disabled
[    2.501036] printk: legacy bootconsole [axi_uart0] disabled
[    2.603599] loop: module loaded

--- exit ---

After debugging, I found that the issue is related to a DHCP problem. The system seems unable to obtain an IP via DHCP, resulting in a timeout and the failure to mount the root filesystem via NFS. The issue logs show the following: It seems like the system is unable to obtain an IP via DHCP, resulting in a timeout and the failure to mount the root filesystem via NFS.


[    2.602331] loop: module loaded
[    2.656500] usbcore: registered new interface driver uas
...
[    2.924755] Sending DHCP requests ...... timed out!
[   79.072854] IP-Config: Retrying forever (NFS root)...
[   79.105593] Sending DHCP requests ...... timed out!
[  166.792537] IP-Config: Retrying forever (NFS root)...
Xu-feng-feng commented 1 day ago

I also tried running this command:

make BOARD=u250 CONFIG=rocket64b2 ROOTFS_URL=192.168.1.100:/sda2/nfsroot  HW_SERVER_ADDR=localhost:3121 ROOTFS=NFS JTAG_BOOT=1 jtag-boot

And the output was as follows:

RISC-V 64, Boot ROM V3.8

OpenSBI v1.4
   ____                    _____ ____ _____
  / __ \                  / ____|  _ \_   _|
 | |  | |_ __   ___ _ __ | (___ | |_) || |
 | |  | | '_ \ / _ \ '_ \ \___ \|  _ < | |
 | |__| | |_) |  __/ | | |____) | |_) || |_
  \____/| .__/ \___|_| |_|_____/|____/_____|
        | |
        |_|

Platform Name             : freechips,rocketchip-vivado
Platform Features         : medeleg
Platform HART Count       : 2
Platform IPI Device       : aclint-mswi
Platform Timer Device     : aclint-mtimer @ 1250000Hz
Platform Console Device   : axi-uart
Platform HSM Device       : ---
Platform PMU Device       : ---
Platform Reboot Device    : ---
Platform Shutdown Device  : ---
Platform Suspend Device   : ---
Platform CPPC Device      : ---
Firmware Base             : 0x80000000
Firmware Size             : 192 KB
Firmware RW Offset        : 0x20000
Firmware RW Size          : 64 KB
Firmware Heap Offset      : 0x27000
Firmware Heap Size        : 36 KB (total), 2 KB (reserved), 9 KB (used), 24 KB (free)
Firmware Scratch Size     : 4096 B (total), 320 B (used), 3776 B (free)
Runtime SBI Version       : 2.0

Domain0 Name              : root
Domain0 Boot HART         : 0
Domain0 HARTs             : 0*,1*
Domain0 Region00          : 0x0000000002000000-0x000000000200ffff M: (I,R,W) S/U: ()
Domain0 Region01          : 0x0000000080020000-0x000000008002ffff M: (R,W) S/U: ()
Domain0 Region02          : 0x0000000080000000-0x000000008001ffff M: (R,X) S/U: ()
Domain0 Region03          : 0x000000000c000000-0x000000000fffffff M: (I,R,W) S/U: (R,W)
Domain0 Region04          : 0x0000000000000000-0xffffffffffffffff M: () S/U: (R,W,X)
Domain0 Next Address      : 0x0000000080200000
Domain0 Next Arg1         : 0x0000000000010080
Domain0 Next Mode         : S-mode
Domain0 SysReset          : yes
Domain0 SysSuspend        : yes

Boot HART ID              : 0
Boot HART Domain          : root
Boot HART Priv Version    : v1.12
Boot HART Base ISA        : rv64imafdcx
Boot HART ISA Extensions  : none
Boot HART PMP Count       : 8
Boot HART PMP Granularity : 2 bits
Boot HART PMP Address Bits: 32
Boot HART MHPM Info       : 0 (0x00000000)
Boot HART MIDELEG         : 0x0000000000000222
Boot HART MEDELEG         : 0x000000000000b109

U-Boot 2022.01-dirty (Nov 29 2024 - 13:53:36 +0800)

CPU:   rv64imafdczicsr_zifencei_zihpm_xrocket
Model: freechips,rocketchip-vivado
DRAM:  14 GiB
MMC:   
Loading Environment from nowhere... OK
In:    uart@60010000
Out:   uart@60010000
Err:   uart@60010000
Net:   No ethernet found.
Hit any key to stop autoboot:  0 
No MMC device available
No ethernet found.
No ethernet found.
=> 

It seems that the system still has trouble detecting the ethernet interface. Could you provide any advice on how to proceed?


eugene-tarassov commented 15 hours ago

The first command is the right one. The problem appears to be either wrong router or wrong Ethernet cable. You need to use 10G SFP+ router and connect it to the first lane of QSFP0 connector using 40G QSFP+ to 4 x 10G SFP+ Breakout Cable.

The message "No ethernet found" is coming from U-Boot, not from the OS. Ethernet in U-Boot is not needed, not supported and not used in this workflow.

Xu-feng-feng commented 3 hours ago

Thank you for your reply and suggestion. I followed your advice and used a 10G SFP+ router, connecting it to the first lane of the QSFP0 connector with a 40G QSFP+ to 4 x 10G SFP+ Breakout Cable.

However, after generating the bitstream for rocket64b2 and downloading it to the XCU250, I noticed that the SFP port light does not turn on. When using a different bitstream (from this source), the SFP port light does turn on, and the router interface shows that the SFP port is connected. But with the rocket64b2 bitstream, the SFP port still shows as not connected.

I’m unsure what might be causing this discrepancy. Could there be something specific to the rocket64b2 configuration that affects the SFP port's behavior?

eugene-tarassov commented 2 hours ago

In this repo, ethernet uses GT channel X1Y44, see board/u250/ethernet-u250.tcl, which matches docs for U250. However, that other example looks like it uses channel X1Y48 for the first lane of the QSFP0 connector. I'm not sure which one is correct, I don't have U250 board, I run tests on a different Alveo board.

Xu-feng-feng commented 2 hours ago

Hi Eugene,

Thank you for your guidance. I understand that the "No ethernet found" message is coming from U-Boot, not the OS. However, I am currently facing this issue where U-Boot is unable to detect Ethernet. Could you provide any insights on how to resolve this?

I have already ensured that I am using the correct 10G SFP+ router and connecting it properly via the 40G QSFP+ to 4 x 10G SFP+ Breakout Cable as per your recommendation.

Is there any additional configuration I need to check or set in U-Boot to enable Ethernet functionality, or could this be related to the specific hardware configuration for the rocket64b2 bitstream?

Any advice would be greatly appreciated!

eugene-tarassov commented 2 hours ago

This ethernet controller is not supported by U-Boot.