Debugging CVA6 APU on a KC705 board

[Juan-Gg]

I am attempting to get the CVA6 APU running on a KC705 board.

What I have done so far:

• Cloned the CVA6 repository, installed the tools and successfully ran smoke-tests.sh
• Ran $make fpga BOARD=kc705, and obtained the ariane_xilinx.mcs file.
• Programmed the kc705’s configuration memory using this file. I am using Vivado 2018.2.
• When pressing the “FPGA Prog” button, after about 3 minutes the “GPIO LEDS” start flashing, four in a counting pattern and the other four in a shifting pattern. No UART activity.
• I prepared an SD card with the Ariane 4.2 image (https://github.com/openhwgroup/cva6-sdk/releases), when powering on with this SD card I could not see any activity on the UART.
• Following the instructions here: https://github.com/openhwgroup/cva6?tab=readme-ov-file#debugging, I tried to debug the design. When I run “openocd -f ariane.cfg”, it does not find the core. I get the following output:

ubuntu@ubuntu-2204:~/Documents$ openocd -f ariane.cfg
Open On-Chip Debugger 0.11.0
Licensed under GNU GPL v2
For bug reports, read
    http://openocd.org/doc/doxygen/bugs.html
DEPRECATED! use 'adapter speed' not 'adapter_khz'
DEPRECATED! use 'adapter driver' not 'interface'
Info : auto-selecting first available session transport "jtag". To override use 'transport select <transport>'.
Info : clock speed 1000 kHz
Error: JTAG scan chain interrogation failed: all ones
Error: Check JTAG interface, timings, target power, etc.
Error: Trying to use configured scan chain anyway...
Error: riscv.cpu: IR capture error; saw 0x1f not 0x01
Warn : Bypassing JTAG setup events due to errors
Error: Unsupported DTM version: 15
Warn : target riscv.cpu examination failed
Info : starting gdb server for riscv.cpu on 3333
Info : Listening on port 3333 for gdb connections
Error: Target not examined yet

Error: Unsupported DTM version: 15

Any ideas on what may be going wrong? What do the LEDs mean? I could not find any documentation on these. Thanks.

[jquevremont]

The SDK is adapted to the Genesys 2 board, which has been selected by OpenHW Group for most of its projects. As of now, the CVA6 team has no resource to provide such support for other boards. OpenHW members are welcome to propose such new projects and contributions.

[jquevremont]

I need to update the README to state the limitation.

(See https://github.com/openhwgroup/cva6-sdk/pull/92)

[Juan-Gg]

But even without sdk support, shouldn't I be able to at least see the core on the JTAG chain, with OpenOCD?

On the README for this repository, it does say:

We currently only provide support for the Genesys 2 board.

But the makefile has support for kc705 & vc707 boards, related scripts for these boards are in corev_apu/fpga/src and corev_apu/fpga/constraints. I understand these are not maintained?

[jquevremont]

Your understanding is right. More contributors to OpenHW projects are welcome to add more such maintenance.

[Juan-Gg]

Here is what I did to get it working. Hope somebody finds it useful.

Generating the bitstream

The LEDs I was mentioning have nothing to do with CVA6. There is no GPIO implemented at the time of writing. Programming the board with the MCS file did not work for some reason, but using the bitstream directly did work and the board booted. This is what the bootrom outputs, if the SD card with the linux image is present, it will boot Linux next.

..Hello World!
init SPI
status: 0x0000000000000025
status: 0x0000000000000025
SPI initialized!
initializing SD... 

In order to look at the Vivado project and be able to see the elaboration results, I changed the target in corev_apu/fpga/Makefile from \$ (mcs) to \$ (bit). This way the Vivado project is not overwritten for MCS generation.

Setting up the JTAG with BSCANE2

Turns out the Genesys 2 FT2232 IC has two JTAG ports broken out, one for the FPGA, one free to use, in this case, for debugging the cva6. The kc705 board, even though having the same IC, only has one JTAG port, used for the FPGA. The cva6 JTAG pins are connected to the FMC connectors, and could be used with an external JTAG adapter. Another option is using Xilinx BSCANE2 IPs, which expose some additional registers in the FPGA’s JTAG, which can be used to debug the processor through the FPGA’s JTAG. This functionality is available as part of the riscv-dbg repository , though a newer version than what is used currently in cva6.

The file “dmi_bscane_tap.sv” can replace “dmi_jtag_tap.sv” to use Xilinx BSCANE2 IPs instead of external pins. As they note “The file is pin-compatible so that by selecting the appropriate file for the target it can be transparently managed without relying on tick defines.”

• Update the riscv-dbg submodule to get the “dmi_bscane_tap.sv”.
• To synthesize correctly it needs an updated version of the common_cells submodule.
• Replace “dmi_jtag_tap.sv” by “dmi_bscane_tap.sv” on the Makefile.
• Massage the included sources until it synthesizes (after this shenanigans the Verilator simulation no longer works in my case).

Note: It may be a good idea to update riscv-dbg and common_cells to more recent versions so this BSCAN option is available by default.

OpenOCD configuration

Then, one can launch OpenOCD (setup, download, install)

I got it to work with the following configuration script, adapted from here.

# Adapted from core-v-mini-mcu-pynq-z2-bscan.cfg

adapter driver ftdi     
adapter speed 1000
transport select jtag

# FT2232HQ Adapter
ftdi_vid_pid 0x0403 0x6010

ftdi_channel 0
ftdi_layout_init 0x0088 0x008b

reset_config none

echo "ftdi setting..."

set _CHIPNAME riscv

jtag newtap $_CHIPNAME cpu -irlen 6 -expected-id 0x43651093

set _TARGETNAME $_CHIPNAME.cpu

target create $_TARGETNAME riscv -chain-position $_TARGETNAME -coreid 0x000

echo "target created..."

#log_output openocd_fpga.log

riscv set_ir idcode 0x09
riscv set_ir dtmcs 0x22
riscv set_ir dmi 0x23

riscv set_reset_timeout_sec 2000
riscv set_command_timeout_sec 2000

echo "setting preferences..."

scan_chain

init

echo "init routine started"

halt

echo "Ready for Remote Connections"

Running OpenOCD:

A sample execution:

$ openocd -f bscan_modified.cfg
xPack Open On-Chip Debugger 0.12.0-01004-g9ea7f3d64-dirty (2023-01-30-15:03)
Licensed under GNU GPL v2
For bug reports, read
    http://openocd.org/doc/doxygen/bugs.html
DEPRECATED! use 'ftdi vid_pid' not 'ftdi_vid_pid'
DEPRECATED! use 'ftdi channel' not 'ftdi_channel'
DEPRECATED! use 'ftdi layout_init' not 'ftdi_layout_init'
ftdi setting...
target created...
setting preferences...
Info : clock speed 1000 kHz
Info : JTAG tap: riscv.cpu tap/device found: 0x43651093 (mfg: 0x049 (Xilinx), part: 0x3651, ver: 0x4)
Info : datacount=2 progbufsize=8
Info : Examined RISC-V core; found 1 harts
Info :  hart 0: XLEN=64, misa=0x800000000014112f
Info : starting gdb server for riscv.cpu on 3333
Info : Listening on port 3333 for gdb connections
init routine started
Ready for Remote Connections
Info : Listening on port 6666 for tcl connections
Info : Listening on port 4444 for telnet connections

Running GDB (This onwards applies to any board):

After OpemOCD is running, one can launch gdb and use it, for example, to examine memory. Here we are looking at 0x10000, the first word in bootrom_64, which matches with the value in bootrom_64.sv.

$ riscv-none-elf-gdb -ex "target remote :3333" 
[...]
(gdb) x /xg 0x00010000
0x10000:    0x3049107300800913

Compiling a baremetal program:

One can take the corev_apu/fpga/src/bootrom as a starting point.

• Copy that directory into another folder.
• Remove all files related to the SD card and device tree, leaving main.c and the UART and SPI drivers.
• In the Makefile:
  • Remove all references to SD card and device tree.
  • Changed riscv64-unknown-elf- by the new riscv-none-elf.
  • Add -march=rv64gc_zba_zbb_zbs_zbc mabi=lp64d to the CFLAGS, after reverse-engineering the cva6.py script, these are set for the cv64a6_imafdc_sv39 configuration and are needed by the new compiler toolchain.
• In main.c: Remove calls to SD card routines and the jump to the First Stage Bootloader that would have been loaded from the SD card.
• In startup.S: Remove anything to do with the device tree.
• In linker.lds, the linker script: Adjust ROM_BASE to 0x80000000, the RAM.
• Add your code in main.c

Then, running make results in an .elf file that can be loaded via gdb.

An example setup can be found here for the time being. Ideally, a similar example should be part of cva6's repository.

Running the baremetal program:

The baremetal program can be executed as follows:

$ riscv-none-elf-gdb -ex "target remote :3333" bootrom_64.elf

[...]

(gdb) load
Loading section .text.init, size 0x88 lma 0x80000000
Loading section .text, size 0x32a lma 0x80000100
Loading section .text.startup, size 0x24 lma 0x8000042a
Loading section .data, size 0x10 lma 0x80000500
Loading section .rodata, size 0x5d lma 0x80000600
Start address 0x000000008000042a, load size 1091
Transfer rate: 12 KB/sec, 218 bytes/write.
(gdb) continue
Continuing.

Terminal output can be seen in any terminal emulator, such as gtkterm. I used dmesg | grep tty to see which ttyUSBx was the board’s UART.

Summary: Once bitstream and .elf are ready:

• Turn the board on, open ariane.xpr in Vivado and connect to the board by opening the hardware manager.
• Load the bitstream with Tools/Program.
• Close the hardware manager to free the JTAG port.
• Launch OpenOCD with its config file: openocd -f config.cfg
• Launch gdb: riscv-none-elf-gdb -ex "target remote :3333" file.elf
• Use gdb: load and continue to run the program. Observe output on terminal.
• You are now Bob's nephew (hopefully).

Adding an Integrated Logic Analyzer (ILA)

[Here] (https://github.com/DvdBerg/cva6/commit/11799a6a082788255ec76ef3e8cb0c341cfff4b2) you can find the changes this guy made to use an ILA.

As a summary:

• Configure an ILA (# ports, port widths etc) in xlnx_ila/tcl/run.tcl
• Generate the IP

  
  $ cd corev_apu/fpga/xilinx/xlnx_ila/
  $ export XILINX_PART=xc7k325tffg900-2
  export XILINX_BOARD=xilinx.com:kc705:part0:1.5
  export BOARD=kc705
  export CLK_PERIOD_NS=20
  $  vivado -mode batch -source tcl/run.tcl

- Include the ILA in the main run.tcl:

read_ip { \ [...] "xilinx/xlnx_ila/xlnx_ila.srcs/sources_1/ip/xlnx_ila/xlnx_ila.xci" \ <------- Add here }

[...]

launch_runs synth_1 wait_on_run synth_1 open_run synth_1

write_debug_probes -force work-fpga/debug_nets.ltx <---------- Add this too

- Instantiate the ILA in the verilog code
```verilog
xlnx_ila ila_test (
        .clk(clk), <--- Watch this, choose the right clock.

        .probe0(your_signal),
        .probe1(your_signal_2),
[...]
    );

• Go for a walk or something while you run synthesis yet again.

Using an Integrated Logic Analyzer (ILA) while running a program with GDB with a shared JTAG port

This one's easy. You don't.

But there is something you can do: Get a program running on the board, debug using the UART and use the JTAG port for the ILA.

• Open the Vivado Hardware Manager, load the bitstream to your FPGA, close the hardware manager.
• Launch OpenOCD and GDB as before, load and continue the program.
• Kill OpenOCD with ctrl+C so the program continues running on the board, but JTAG is now free.
• Open the Hardware Manager again, and watch the ILA waveforms.

To allow you to control the program flow from the UART, i.e. to start a transaction you wish to trigger on with the ILA, a function to receive on the UART comes in handy.

One can add something like this to uart.c

#define UART_LINE_STATUS_BIT_DATA_READY (1 << 0)

char uart_receive_char_blocking()
{
    while(!(read_reg_u8(UART_LINE_STATUS) & UART_LINE_STATUS_BIT_DATA_READY));
    return  read_reg_u8(UART_RBR); // Receive buffer register.
}

[elialaz]

Hi juan,

Could you plis share a baremetal c project example? I'm quite stuck and I cannot run anything successfully unfortunately.

Thanks

[Juan-Gg]

@elialaz, I have added a link on my previous post. I hope it works for you.

[elialaz]

Thanks very much!