Bringup FreeRTOS on FireSim

[dhand-galois]

This will likely involve multiple steps, some of which can be done concurrently, so this can become a global tracking issue.

• [x] #87 Build the 32-bit FireSim image and run a bare metal test
• [x] Bringup FreeRTOS w/ only UART
• [x] Port iceNIC driver to FreeRTOS
• [x] Port iceblk driver to FreeRTOS
• [x] Bringup FreeRTOS w/ all devices working

[dhand-galois]

@podhrmic JoeK mentioned you as someone that could help with porting the drivers to FreeRTOS. Are you able to take that on?

[podhrmic]

Sure thing. Do you have a system description / example of drivers for linux somewhere? Also, this is something @waylon531 might be interested in - if so, I am happy to let you have it.

[jrtc27]

https://github.com/firesim/iceblk-driver and https://github.com/firesim/icenet-driver

[podhrmic]

That is helpful. Regarding the virtual UART in FireSim - is there a description of it somewhere (or a linux driver)?

[jrtc27]

If this is the sifive UART, BBL has a minimal driver in machine/uart.c (not uart16550.c), FreeBSD has a driver in sys/riscv/sifive/sifive_uart.c, Linux in drivers/tty/serial/sifive.c and https://sifive.cdn.prismic.io/sifive/4d063bf8-3ae6-4db6-9843-ee9076ebadf7_fe310-g000.pdf Chapter 19 (page 68) has the specification.

[dhand-galois]

FYI on FireSim's UART - the UART device actually communicates with another UART device on the FPGA. That "receiver" has a hard-coded divisor / baud rate. Based on our current builds, this number is derived from the 100MHz "effective" bus frequency with a baud rate of 3686400 to obtain a divisor of 27.

[kiniry]

👀

/me no comment

[dhand-galois]

FYI, there is a SiFive UART driver for FreeRTOS, but it relies heavily on SiFive's metal platform: https://github.com/sifive/Amazon-FreeRTOS/blob/master/vendors/sifive/boards/hifive1_rev_b/aws_demos/application_code/sifive_code/freedom-metal/src/uart.c https://github.com/sifive/Amazon-FreeRTOS/blob/master/vendors/sifive/boards/hifive1_rev_b/aws_demos/application_code/sifive_code/freedom-metal/src/drivers/sifive_uart0.c

May or may not be helpful.. looking at it, I find the FreeBSD or Linux driver easier to read but they're all very similar.

[dhand-galois]

Per #87, there is now a P1 AFI available for testing binaries.

I've also created a new minimal_cloudgfe with a pre-built AFI for the P1. Instructions here (waiting on some improvements to local build flow before merging in completely)

[podhrmic]

Do you have a device tree for the P1? I would like to find out where are icenet and iceblock located

[dhand-galois]

(Just for general info) The DTS is generated dynamically when you build the design in chipyard or FireSim. It's located at firesim/sim/generated-src/f1/FireSim-WithNIC_DDR3FRFCFSLLC4MB_FireSimCloudGFERocketP1Config-F90MHz_BaseF1Config/firesim.firesim.FireSim.WithNIC_DDR3FRFCFSLLC4MB_FireSimCloudGFERocketP1Config.dts.

To save the trouble of fully setting up FireSim:

/dts-v1/;

/ {
    #address-cells = <1>;
    #size-cells = <1>;
    compatible = "galois,gfe-dev";
    model = "galois,gfe";
    L21: aliases {
        serial0 = &L14;
    };
    L16: chosen {
        bootargs = "earlyprintk console=hvc0 earlycon=sbi";
        stdout-path = &L14;
    };
    L20: L17: cpus {
        #address-cells = <1>;
        #size-cells = <0>;
        timebase-frequency = <1000000>;
        L6: cpu@0 {
            clock-frequency = <100000000>;
            compatible = "sifive,rocket0", "riscv";
            d-cache-block-size = <64>;
            d-cache-sets = <64>;
            d-cache-size = <4096>;
            device_type = "cpu";
            hardware-exec-breakpoint-count = <1>;
            i-cache-block-size = <64>;
            i-cache-sets = <64>;
            i-cache-size = <4096>;
            next-level-cache = <&L7>;
            reg = <0x0>;
            riscv,isa = "rv32imac";
            riscv,pmpregions = <8>;
            status = "okay";
            timebase-frequency = <1000000>;
            L4: interrupt-controller {
                #interrupt-cells = <1>;
                compatible = "riscv,cpu-intc";
                interrupt-controller;
            };
        };
    };
    L9: memory@80000000 {
        device_type = "memory";
        reg = <0x80000000 0x80000000>;
    };
    L19: soc {
        #address-cells = <1>;
        #size-cells = <1>;
        compatible = "galois,gfe-soc", "simple-bus";
        ranges;
        L13: blkdev-controller@40015000 {
            compatible = "ucbbar,blkdev";
            interrupt-parent = <&L1>;
            interrupts = <2>;
            reg = <0x40015000 0x1000>;
            reg-names = "control";
        };
        L7: cache-controller@2010000 {
            cache-block-size = <64>;
            cache-level = <2>;
            cache-sets = <1024>;
            cache-size = <524288>;
            cache-unified;
            compatible = "sifive,inclusivecache0", "cache";
            next-level-cache = <&L9>;
            reg = <0x2010000 0x1000>;
            reg-names = "control";
            sifive,mshr-count = <7>;
        };
        L2: clint@10000000 {
            compatible = "riscv,clint0";
            interrupts-extended = <&L4 3 &L4 7>;
            reg = <0x10000000 0x10000>;
            reg-names = "control";
        };
        L3: debug-controller@0 {
            compatible = "sifive,debug-013", "riscv,debug-013";
            interrupts-extended = <&L4 65535>;
            reg = <0x0 0x1000>;
            reg-names = "control";
        };
        L0: error-device@3000 {
            compatible = "sifive,error0";
            reg = <0x3000 0x1000>;
        };
        L15: ice-nic@62100000 {
            compatible = "ucbbar,ice-nic";
            interrupt-parent = <&L1>;
            interrupts = <4 5>;
            reg = <0x62100000 0x1000>;
            reg-names = "control";
        };
        L1: interrupt-controller@c000000 {
            #interrupt-cells = <1>;
            compatible = "riscv,plic0";
            interrupt-controller;
            interrupts-extended = <&L4 11>;
            reg = <0xc000000 0x4000000>;
            reg-names = "control";
            riscv,max-priority = <7>;
            riscv,ndev = <5>;
        };
        L12: mmint@2000000 {
            compatible = "ssith,mmint";
            interrupts-extended = <&L1 1>;
            reg = <0x2000000 0x100>;
            reg-names = "control";
        };
        L11: rom@70000000 {
            compatible = "sifive,rom0";
            reg = <0x70000000 0x10000>;
            reg-names = "mem";
        };
        L14: serial@62300000 {
            compatible = "sifive,uart0";
            interrupt-parent = <&L1>;
            interrupts = <3>;
            reg = <0x62300000 0x1000>;
            reg-names = "control";
        };
    };
};

[podhrmic]

FYI we have the UART FreeRTOS drivers updated to work with Firesim P1 - later today I will update and merge the respective PR. I will look at the network interface next, but given my schedule (meetings this afternoon and OOO tomorrow), I would say realistically we should count on having only UART this sprint.

[austinharris]

FWIW, it would be a significant help to the Michigan team if we can get the network interface on FreeRTOS by mid next week.

[podhrmic]

Right on, I think that is a realistic timeline.

[dhand-galois]

To keep this issue up to date, @podhrmic has a build of FreeRTOS successfully printing to the UART. https://github.com/DARPA-SSITH-Demonstrators/FreeRTOS-10.0.1/pull/13

There was a hardware configuration mismatch with the timer being set to count 100x slower than the one on the GFE. A new AFI will be available tomorrow that fixes that issue.

[podhrmic]

@austinharris FYI the NIC is now working - use the setup as described here: https://github.com/DARPA-SSITH-Demonstrators/BESSPIN-CloudGFE/blob/firesim-add-p1/FireSim/minimal_cloudgfe.md

(some small HW changes in the NIC and CLINT were needed), and then for SW use https://github.com/DARPA-SSITH-Demonstrators/FreeRTOS-10.0.1 and develop branch

The main_tcp/main_udp/main_peekpoke are all good examples.

Don't forget to build with BSP=aws to specify that you are building for CloudGFE

Let me know if you have any questions.

[austinharris]

@taru-verma please test this out.

[dhand-galois]

@podhrmic good to close this issue?

[podhrmic]

Yes, I thought I closed from gitkraken, but apparently not...