search

openhwgroup / cva6-sdk

CVA6 SDK containing RISC-V tools and Buildroot
59 stars 64 forks source link

Linux Kernel Cannot Print where Bootrom and BBL can #39

Open Kendidi opened 4 years ago

Kendidi commented 4 years ago

Hi,

I tried to load and run Bootrom, BBL and Linux Kernel on FPGA.

Bootrom and BBL can print on UART console without any issue, but the Linux Kernel cannot, even it went pass through a few places in setup.c and init.c where it should have printed something on the console. I think the ariane.dts in bootrom should be OK, otherwise BBL may not be able to print.

I wonder: 1) What may be the issue? Where should I look at? 2) How does the UART/console get initialized normally so that functions like _prinfo( ) and printk( ) can print?

Thanks a lot in advance!

Jbalkind commented 3 years ago

Ariane master is probably a good call. There have been a few important changes recently though (mainly the PMPs and 32 bit support) and so I'm not sure what state everything is in.

If it's single core, the core can just be waiting for a timer that it set for itself. It's possible that that wfi is just the OS's idle loop. How did you check the instruction?

Kendidi commented 3 years ago

Thanks @Jbalkind.

I used GDB to run the code. I can see printout from Bootrom and from bbl. But nothing from Linux. I stopped GDB and it showed:

(gdb) c
Continuing.
^C
Program received signal SIGINT, Interrupt.
0xffffffe000000124 in ?? ()
(gdb)

I used riscv64-unknown-elf-objdump to disassemble vmlinux and it shows: ffffffe00000007a : ...... ...... ffffffe000000120: 2aa0206f j ffffffe0000023ca ffffffe000000124: 10500073 wfi ffffffe000000128: bff5 j ffffffe000000124 <relocate+0xaa>

Jbalkind commented 3 years ago

Looks like that corresponds with this (assuming kernel 5.1): https://elixir.bootlin.com/linux/v5.1/source/arch/riscv/kernel/head.S#L162

Weird place to be stuck. Maybe try disabling CONFIG_SMP if you're not using it?

Kendidi commented 3 years ago

Yes, it's 5.1 After CONFIG_SMP is disabled in config file. It still stuck at instruction "wfi". Address of the instructions shifted though. 

(gdb) c
Continuing.
^C
Program received signal SIGINT, Interrupt.
0xffffffe0000000e0 in ?? ()
(gdb)

ffffffe0000000de:       0001                    nop
ffffffe0000000e0:       10500073                wfi
ffffffe0000000e4:       bff5                    j       ffffffe0000000e0 <relocate+0x66>
Jbalkind commented 3 years ago

If you look at the comments above where that's put in the trap vector here: https://elixir.bootlin.com/linux/v5.1/source/arch/riscv/kernel/head.S#L112 then you can see it's intended to help debug a particular error case. Can you maybe try and get the trap pc from gdb and see what it was executing before this happened?

Honestly I find this case you're stuck in pretty weird. I would again encourage you to use opensbi if there's any possibility of that for your use case.

Kendidi commented 3 years ago

Not sure how to get the trap PC from GDB since BT doesn't in current environment.

I have considered replacing bbl with opensbi. I just don't know where to start. Is there any documents/website demonstrate a good way to do it? Thanks.

Kendidi commented 3 years ago

With Kernel v5.3 and bbl from ariane-sdk, it stuck in memset() instead.

Kendidi commented 3 years ago

@jctullos

In my situation, the following are required for Kernel to print messages onto the console. Somehow SMP needs to be off for the other changes to be effective.

CONFIG_SMP=n

CONFIG_HVC_DRIVER=y
CONFIG_SERIAL_EARLYCON_RISCV_SBI=y

CONFIG_CMDLINE="earlyprintk console=hvc earlycon=sbi"

Unfortunately, there are still some issues that forbid it to boot all the way. Have you encountered that before? Thanks.

[    0.000000] OF: fdt: Ignoring memory range 0x80000000 - 0x80200000
[    0.000000] Linux version 5.5.0 (root@aie-Machine) (gcc version 8.2.0 (GCC))0
[    0.000000] earlycon: sbi0 at I/O port 0x0 (options '')
[    0.000000] printk: bootconsole [sbi0] enabled
[    0.000000] initrd not found or empty - disabling initrd
[    0.000000] Zone ranges:
[    0.000000]   DMA32    [mem 0x0000000080200000-0x00000000bfffffff]
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080200000-0x00000000bfffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080200000-0x00000000bfffffff]
[    0.000000] software IO TLB: mapped [mem 0xbb1fc000-0xbf1fc000] (64MB)
[    0.000000] elf_hwcap is 0x112d
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 258055
[    0.000000] Kernel command line: earlyprintk console=hvc earlycon=sbi
[    0.000000] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
[    0.000000] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes, l)
[    0.000000] Sorting __ex_table...
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] BUG: Bad page state in process swapper  pfn:80ce4
[    0.000000] page:ffffffe03f02c1e0 refcount:1 mapcount:0 mapping:0000000000000
[    0.000000] raw: 0000000000000000 ffffffe03f02c1e8 ffffffe03f02c1e8 000000000
[    0.000000] raw: 0000000000000000 0000000000000000 00000001ffffffff
[    0.000000] page dumped because: nonzero _refcount
[    0.000000] Modules linked in:
[    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 5.5.0 #5
[    0.000000] Call Trace:
[    0.000000] [<ffffffe0005f1158>] walk_stackframe+0x0/0xa0
[    0.000000] [<ffffffe0005f133a>] show_stack+0x2a/0x34
[    0.000000] [<ffffffe000938f46>] dump_stack+0x20/0x28
[    0.000000] [<ffffffe00065a8fc>] bad_page+0xe0/0xfe
[    0.000000] [<ffffffe00065a952>] free_pages_check_bad+0x38/0x7a
[    0.000000] [<ffffffe00065af62>] __free_pages_ok+0xa4/0x4c2
[    0.000000] [<ffffffe00065c220>] __free_pages.part.142+0x1a/0x22
[    0.000000] [<ffffffe00065c2e4>] __free_pages_core+0x94/0xa0
[    0.000000] [<ffffffe000006bbe>] memblock_free_pages+0x12/0x1a
[    0.000000] [<ffffffe0000083f8>] memblock_free_all+0x14e/0x1ae
[    0.000000] [<ffffffe0000026a0>] mem_init+0x2a/0x38
[    0.000000] [<ffffffe0000008d2>] start_kernel+0x1f6/0x394
[    0.000000] [<ffffffe0000000bc>] clear_bss_done+0x3c/0x40
[    0.000000] Disabling lock debugging due to kernel taint
[    0.000000] BUG: Bad page state in process swapper  pfn:80ce8
[    0.000000] page:ffffffe03f02c2c0 refcount:1 mapcount:0 mapping:0000000000000
[    0.000000] raw: 0000000000000000 ffffffe03f02c2c8 ffffffe03f02c2c8 000000000
[    0.000000] raw: 0000000000000000 0000000000000000 00000001ffffffff
[    0.000000] page dumped because: nonzero _refcount
......
......
jctullos commented 3 years ago

@Kendidi

Hey sorry I didn't realize the progress you've made! I should have said one of the other configure options I had was SMP off. I've seen it in the past where there's a bug that if only one core is running and SMP is on, it hangs.

What commit are you building the Ariane processor from on the CVA6 repo?

jctullos commented 3 years ago

So here's my suggestion also. If you're using the OpenPiton or ESP build, and you don't need PMP, then stick to whichever Ariane version they have linked. There's an issue with PMP that I believe isn't getting captured up at the top level builds for those two, which was why the ESP folks added an issue to track it.

If you don't need PMP, then keep the Ariane at the same commit as the OpenPiton/ESP builds. Also keep Linux on the 5.1 or 5.3, it says in your boot logs that you're loading Linux 5.5.

The problem with this is that the Linux defconfig in the Ariane SDK is based on Linux 5.1. Same with the riscv-pk (bbl) is older. So there were a bunch of changes that happened to go to 5.5, and I believe riscv-pk had been updated as well. Some of the references in the bbl code, or the encoding.h, get updated with new values that are used in a newer version of Linux. So having 2 different versions of the bbl and Linux can work, but sometimes going too far apart it will mess with it.

If you do need PMP, with the vcu128 you might be out of luck right now. There's a post on the CVA6 issues where a guy was able to get a VCU108 working and he has a repo. You could follow his changes to get your VCU128 going. I did the same for the VCU118 but was having some uart issues. And I got the vc707 working so I've switched over to it.

Kendidi commented 3 years ago

@jctullos

Thank you very much for your advice!

What commit are you building the Ariane processor from on the CVA6 repo?

I have been using Ariane CVA6 commit #453 since Thursday because I think it's prior to PMP support and it was recommended by the ESP folks.

it says in your boot logs that you're loading Linux 5.5. Yeah, I didn't see any improvement on the issues with 5.1 and 5.3, so I tried 5.5 to see if there'll be difference.

So what setup/config you have now that will allow Kernel to boot successfully? Are you still having SMP turned off and with the other 3 config option set?

jctullos commented 3 years ago

Can you link it here the commit url for CVA6?

Using the default Ariane SDK build (probably worth it to wipe it and start with a fresh repo) the only changes I made were to the Linux defconfig are:

The changes I did make to the Ariane SDK linux_defconfig are:

CONFIG_DEFAULT_HOSTNAME="ariane-fpga"

CONFIG_CROSS_MEMORY_ATTACH is not set

CONFIG_NAMESPACES=y CONFIG_BLK_DEV_INITRD=y CONFIG_INITRAMFS_SOURCE="${BR_BINARIES_DIR}/rootfs.cpio" CONFIG_EMBEDDED=y CONFIG_SMP=y CONFIG_HZ_100=y CONFIG_CMDLINE="earlyprintk console=hvc earlycon=sbi" CONFIG_MODULES=y

CONFIG_BLK_DEV_BSG is not set

CONFIG_PARTITION_ADVANCED=y

CONFIG_COMPACTION is not set

CONFIG_NET=y CONFIG_PACKET=y CONFIG_UNIX=y CONFIG_TLS=y CONFIG_INET=y CONFIG_IP_MULTICAST=y

CONFIG_WIRELESS is not set

CONFIG_DEVTMPFS=y CONFIG_DEVTMPFS_MOUNT=y CONFIG_VIRTIO_BLK=y CONFIG_EEPROM_AT24=y CONFIG_NETDEVICES=y CONFIG_VXLAN=y

CONFIG_LOWRISC_DIGILENT_100MHZ=y

CONFIG_XILINX_EMACLITE=y

CONFIG_MDIO_BCM_UNIMAC=y

CONFIG_MDIO_BITBANG=y

CONFIG_MDIO_BUS_MUX_GPIO=y

CONFIG_MDIO_BUS_MUX_MMIOREG=y

CONFIG_MDIO_BUS_MUX_MULTIPLEXER=y

CONFIG_MDIO_GPIO=y

CONFIG_REALTEK_PHY=y

CONFIG_WLAN is not set

CONFIG_INPUT_KEYBOARD is not set

CONFIG_INPUT_MOUSE is not set

CONFIG_VT is not set

CONFIG_SERIAL_8250=y CONFIG_SERIAL_8250_CONSOLE=y CONFIG_SERIAL_OF_PLATFORM=y CONFIG_SERIAL_UARTLITE=y CONFIG_HVC_RISCV_SBI=y CONFIG_VIRTIO_CONSOLE=y

CONFIG_HW_RANDOM is not set

CONFIG_I2C=y CONFIG_I2C_OCORES=y

CONFIG_SPI=y

CONFIG_SPI_XILINX=y

CONFIG_GPIOLIB=y CONFIG_GPIO_SYSFS=y CONFIG_GPIO_XILINX=y CONFIG_POWER_RESET=y CONFIG_POWER_RESET_GPIO_RESTART=y CONFIG_USB=y CONFIG_USB_XHCI_HCD=y CONFIG_USB_EHCI_HCD=y

CONFIG_MMC=y

CONFIG_MMC_SPI=y

CONFIG_NEW_LEDS=y CONFIG_LEDS_CLASS=y CONFIG_LEDS_PWM=y CONFIG_LEDS_TRIGGERS=y CONFIG_LEDS_TRIGGER_HEARTBEAT=y CONFIG_LEDS_TRIGGER_PANIC=y CONFIG_VIRTIO_MMIO=y

CONFIG_IOMMU_SUPPORT is not set

CONFIG_PWM=y CONFIG_SIFIVE_PLIC=y CONFIG_EXT3_FS=y

CONFIG_PROC_PAGE_MONITOR is not set

CONFIG_TMPFS=y

CONFIG_MISC_FILESYSTEMS is not set

CONFIG_NFS_FS=y CONFIG_NFS_V4=y CONFIG_NFS_SWAP=y CONFIG_NFS_V4_1=y CONFIG_NFS_V4_2=y CONFIG_CRYPTO_ECHAINIV=y

CONFIG_CRYPTO_HW is not set

CONFIG_PRINTK_TIME=y CONFIG_STRIP_ASM_SYMS=y CONFIG_DEBUG_SECTION_MISMATCH=y CONFIG_STACKTRACE=y

So all I did was remove the ethernet items they have listed. I even kept the SMP it looks like. But this allowed the VCU118 to build and boot, so it should work for you as well.

Kendidi commented 3 years ago

By the way, I recall if I set the following options in the ESP environment, then Kernel will crash during boot the same way as my Ariane now does.

CONFIG_HVC_DRIVER=y
CONFIG_SERIAL_EARLYCON_RISCV_SBI=y

CONFIG_CMDLINE="earlyprintk console=hvc earlycon=sbi"

Have you experienced that? To me, seems like these options help printing messages on the console, but somehow will generate issues later on on both Ariane and ESP platforms.

Kendidi commented 3 years ago

Can you link it here the commit url for CVA6?

https://github.com/openhwgroup/cva6/pull/453/files

Thank you very much for the config option list! I will try it.

Kendidi commented 3 years ago

@jctullos

No luck so far. I downloaded SDK. Replaced default linux_defconfig with yours and compiled bbl.

Below is the DST file I am using. The clock frequency we are using is 40MHz and time-base frequency is 20MHz. Our design doesn't support ethernet. It shouldn't matter if we keep CONFIG_NET enabled. Right?

/dts-v1/;

/ {
  #address-cells = <2>;
  #size-cells = <2>;
  compatible = "eth,ariane-bare-dev";
  model = "eth,ariane-bare";
  chosen {
    stdout-path = "/soc/uart@10000000:115200";
  };
  cpus {
    #address-cells = <1>;
    #size-cells = <0>;
    timebase-frequency = <20000000>; // 20 MHz
    CPU0: cpu@0 {
      clock-frequency = <40000000>; // 40 MHz
      device_type = "cpu";
      reg = <0>;
      status = "okay";
      compatible = "eth, ariane", "riscv";
      riscv,isa = "rv64imafdc";
      mmu-type = "riscv,sv39";
      tlb-split;
      // HLIC - hart local interrupt controller
      CPU0_intc: interrupt-controller {
        #interrupt-cells = <1>;
        interrupt-controller;
        compatible = "riscv,cpu-intc";
      };
    };
  };
  memory@80000000 {
    device_type = "memory";
    reg = <0x0 0x80000000 0x0 0x40000000>;
  };
//   leds {
//     compatible = "gpio-leds";
//     heartbeat-led {
//       gpios = <&xlnx_gpio 1 0>;
//       linux,default-trigger = "heartbeat";
//       retain-state-suspended;
//     };
//   };
  L26: soc {
    #address-cells = <2>;
    #size-cells = <2>;
    compatible = "eth,ariane-bare-soc", "simple-bus";
    ranges;
    clint@2000000 {
      compatible = "riscv,clint0";
      interrupts-extended = <&CPU0_intc 3 &CPU0_intc 7>;
      reg = <0x0 0x2000000 0x0 0xc0000>;
      reg-names = "control";
    };
    PLIC0: interrupt-controller@c000000 {
      #address-cells = <0>;
      #interrupt-cells = <1>;
      compatible = "riscv,plic0";
      interrupt-controller;
      interrupts-extended = <&CPU0_intc 11 &CPU0_intc 9>;
      reg = <0x0 0xc000000 0x0 0x4000000>;
      riscv,max-priority = <7>;
      riscv,ndev = <30>;
    };
    debug-controller@0 {
      compatible = "riscv,debug-013";
      interrupts-extended = <&CPU0_intc 65535>;
      reg = <0x0 0x0 0x0 0x1000>;
      reg-names = "control";
    };
    uart@10000000 {
      compatible = "ns16750";
      reg = <0x0 0x10000000 0x0 0x1000>;
      clock-frequency = <40000000>;
      current-speed = <115200>;
      interrupt-parent = <&PLIC0>;
      interrupts = <1>;
      reg-shift = <2>; // regs are spaced on 32 bit boundary
      reg-io-width = <4>; // only 32-bit access are supported
    };
    timer@18000000 {
      compatible = "pulp,apb_timer";
      interrupts = <0x00000004 0x00000005 0x00000006 0x00000007>;
      reg = <0x00000000 0x18000000 0x00000000 0x00001000>;
      interrupt-parent = <&PLIC0>;
      reg-names = "control";
    };
//     xps-spi@20000000 {
//       compatible = "xlnx,xps-spi-2.00.b", "xlnx,xps-spi-2.00.a";
//       #address-cells = <1>;
//       #size-cells = <0>;
//       interrupt-parent = <&PLIC0>;
//       interrupts = < 2 2 >;
//       reg = < 0x0 0x20000000 0x0 0x1000 >;
//       xlnx,family = "kintex7";
//       xlnx,fifo-exist = <0x1>;
//       xlnx,num-ss-bits = <0x1>;
//       xlnx,num-transfer-bits = <0x8>;
//       xlnx,sck-ratio = <0x4>;
// 
//       mmc@0 {
//         compatible = "mmc-spi-slot";
//         reg = <0>;
//         spi-max-frequency = <12500000>;
//         voltage-ranges = <3300 3300>;
//         disable-wp;
//       };

      // mmc-slot@0 {
      //   compatible = "fsl,mpc8323rdb-mmc-slot", "mmc-spi-slot";
      //   reg = <0>;  //Chip select 0
      //   spi-max-frequency = <12500000>;
      //   voltage-ranges = <3300 3300>;
      //   //interrupts = < 2 2 >;
      //   //interrupt-parent = <&PLIC0>;
      // };
//     };
//     eth: lowrisc-eth@30000000 {
//       compatible = "lowrisc-eth";
//       device_type = "network";
//       interrupt-parent = <&PLIC0>;
//       interrupts = <3 0>;
//       local-mac-address = [00 18 3e 02 e3 7f]; // This needs to change if more than one GenesysII on a VLAN
//       reg = <0x0 0x30000000 0x0 0x8000>;
//     };
    xlnx_gpio: gpio@40000000 {
      #gpio-cells = <2>;
      compatible = "xlnx,xps-gpio-1.00.a";
      gpio-controller ;
      reg = <0x0 0x40000000 0x0 0x10000 >;
      xlnx,all-inputs = <0x0>;
      xlnx,all-inputs-2 = <0x0>;
      xlnx,dout-default = <0x0>;
      xlnx,dout-default-2 = <0x0>;
      xlnx,gpio-width = <0x8>;
      xlnx,gpio2-width = <0x8>;
      xlnx,interrupt-present = <0x0>;
      xlnx,is-dual = <0x1>;
      xlnx,tri-default = <0xffffffff>;
      xlnx,tri-default-2 = <0xffffffff>;
    };
  };
};
jctullos commented 3 years ago

Which build are you using? OpenPiton or ESP?

Kendidi commented 3 years ago

CVA6 commit #453. Which one are you using?

jctullos commented 3 years ago

So you customized the CVA6 build for a VCU128? You're not relying on ESP/OpenPiton?

FYI I didn't get my VCU118 working on the CVA6, it builds but the onboard uart wasn't working for me. Could you upload your build to github for the vcu128 for CVA6? If yours works I could take a look at why my uart doesnt.

Kendidi commented 3 years ago

We thought CVA6 is the simplest therefore should be easiest to work with.

I only have info on the device tree. I do not have source code to the build.

Kendidi commented 3 years ago

For ESP builds, which of the following is being using?

/esp/third-party/ariane/bootrom
/esp/third-party/ariane/fpga/src/bootrom
/esp/third-party/ariane/openpiton/bootrom
/esp/soft/ariane/bootrom
paulmnt commented 3 years ago

/esp/soft/ariane/bootrom

If you look at the Make targets in /utils/Makefile, you should be able to navigate all the dependencies. Also, adding the flag -n to any make target will give you the exact command lines used.

e.g.

make soft -n

The other folders exist because they are part of the Ariane's repository.

The ESP-Ariane bootloader /esp/soft/ariane/bootrom is very similar to the original one in /esp/third-party/ariane/fpga/src/bootrom, but we removed code that is unnecessary for ESP and applied minor patches to support the APB UART from GRLIB.

Kendidi commented 3 years ago

Cool! Thank you @paulmnt !!

jimaandro commented 3 months ago

@Jbalkind

Thank you for your advice! Yes I am using FPGA - Xilinx VCU128. It's pretty updated I think. Which version would you recommend? Should we used the latest in repository - "master" or the last release - Ariane 4.2?

Hello @Kendidi I am trying to do something similar. I want to boot Linux on OpenPiton, using VCU128. My question is, how did you conncet JTAG in order to use openOCD? What constraints did you use, and did you change something in verilog files? And what .cfg file did you used for openOCD? Also did you change the constraints for UART? Because UART and JTAG use the same port on VCU128