sizeAndPermissions due to memory change

[HU90m]

When running the CHERIoT RTOS, I'm hitting an error that looks to be a bug in the hardware.

A global constant is being read back as the wrong value when running on FPGA.

The CHERIoT RTOS build:

git clone --recurse-submodules https://github.com/HU90m/cheriot-rtos/tree/sonata
cd sonata
nix develop github:lowRISC/lowrisc-nix#cheriot
xmake f -P examples/01.hello_world/ --board=sonata \
    --debug-scheduler=y --debug-locks=y --debug-cxxrt=y --debug-loader=y --debug-token_library=y
xmake -P examples/01.hello_world

The image is found at build/cheriot/cheriot/release/hello_world.

The tail of the UART log when running on the FPGA:

loader: Error handler for compartment is 0)
loader: Error handler for compartment is 0)
loader: Error handler for compartment is 24)
loader: First pass to find sealing key imports
loader: Priv Compartment 0, addr: 0x100f80, size: 0x1d20
loader: Thing: 0x81000000; 0x66120000
../../sdk/core/loader/types.h:956 Invariant failure in size
Unused bits in sizeAndPermissions are not zero, field contains 0x66120000

This field value should be 0xc0001000 and it is in the ELF file, if you look at the contents of section scheduler_code (address 0x100f90). But for some reason at run time it is a different value, even when checked right at the beginning of the loader_entry_point.

When running the same image on the Verilator simulator, this value is correct and the simulation continues past it, which is rather odd.

My best guess at the moment is that there is something wrong the bus or SRAM access.

The simulator fails after this point for a different reason, which I haven't investigated:

loader: Skipping sealing type import
loader: Building mmio capability 0x107b30 + 0x174d0 (permissions: G RWc-m- -- ---)
loader: Rounding heap (0x107b30--0x11f000) region
loader: Applying mask 0xffffff00 to entry.size
loader: Rounding heap entry.size down from 0x174d0 to 0x17400 (rounded up to 0x17500)
loader: Assigning rounded heap (in 0x107c00--0x11f000) to the allocator
loader: Heap: 0x107c00 (v:1 0x107c00-0x11f000 l:0x17400 o:0x0 p: G RWcgm- -- ---)
loader: Building mmio capability 0x107ae8 + 0x10 (permissions: G RWc--- -- ---)
../../sdk/core/loader/boot.cc:467 Invariant failure in operator()
0x107ae8 is not in the MMIO range

You may want to use https://github.com/lowRISC/sonata-system/pull/53 when running longer simulations.

[marnovandermaas]

An initial thought, the bus in the Verilator simulation is the same as in the FPGA so that is unlikely the problem.

[johngt]

CC @alees24

[marnovandermaas]

It looks like this issue has been resolved in the latest version of the main branch. In conjunction with the following fix: https://github.com/lowRISC/sonata-system/pull/61 Hugo can provide more details but I'm closing this for now.

[HU90m]

It looks like this issue has been resolved in the latest version of the main branch. In conjunction with the following fix: #61 Hugo can provide more details but I'm closing this for now.

I have no idea what fixed this problem; a lot has changed in the repository since the last rebase. I'd like to get CI running on this soon-ish, so we can catch it if it does happen to pop up again.

61 was an unrelated known problem with the timer that stopped the simulation leaving the RTOS scheduler

[HU90m]

This little fella popped up again.

loader: Error handler for compartment is -1
loader: Error handler for compartment is 216
loader: First pass to find sealing key imports
loader: Creating sealing key 0x10
../sdk/core/loader/types.h:956 Invariant failure in size
Unused bits in sizeAndPermissions are not zero, field contains 0xc4000000

The branches

• sonata-system/size-perm-bug-256
• cheriot-rtos/size-perm-bug-256

The build

pushd tests
xmake f --board=sonata --debug-scheduler=y --debug-locks=y --debug-cxxrt=y --debug-loader=y --debug-token_library=y
xmake

Loaded onto the FPGA with the sonata system's ./util/mem_helper.sh.

[HU90m]

I traced the re-emergance was actually a different issue. I traced it back to the RTOS not being able to handle very large MMIO regions.

[marnovandermaas]

Good work Hugo!