CHERI-RISC-V swap panic

[nwf]

Here's a panic in swp_pager_async_iodone; the fault address, 0xffffffc0005ac242, is the cseal ca1, ca1, ca2 instruction therein.

This is

FreeBSD 14.0-CURRENT #1 dev-n252990-855b453a24b2f-dirty: Tue Jun  7 00:13:15 UTC 2022
    daemon@aa131183ba09:/cheri/build/mainline/cheribsd-riscv64-purecap-build/cheri/source/mainline/cheribsd/riscv.riscv64c/sys/CHERI-QEMU riscv
clang version 13.0.0 (https://github.com/CTSRD-CHERI/llvm-project.git cd33be315c091a9542a78e67017c7827705af0aa)

which is really 6533f358f80aa8dbde80b13851210d4082911a3c with unrelated (I hope) changes atop it. Running in qemu using the default disk image, with the default 2 GiB /dev/gpt/swap.

What more can I add that would be useful?

ra = 0xffffffc0005ac204                                                                                                                                                                                                                      
sp = 0xffffffc000d31570 [rwxRW,0x0000000000000000-0xffffffffffffffff]                                                                                                                                                                        
gp = 0x0000000000000000                                                                                                                                                                                                                      
tp = 0xffffffc000a15a40 [rwxRW,0xde80000000000000-0x1a80000000000000] (invalid,sealed)                                                                                                                                                       
t[0] = 0x0000000000000001                                  
t[1] = 0x000000000000002d                                  
t[2] = 0x0000000000006000                                  
t[3] = 0x0000000000000000                                  
t[4] = 0x000000000000ffff                                  
t[5] = 0x0000000000000000                                  
t[6] = 0x0000000000000002                                  
s[0] = 0xffffffc000d31600                                  
s[1] = 0x2ea098ffffffffff                                  
s[2] = 0xffffffc000ad9fb0                                                                                             
s[3] = 0xffffffd024024838                                  
s[4] = 0xffffffc09ab9b940                       
s[5] = 0xffffffd003c02000                                  
s[6] = 0xffffffd1f83dbb40                                  
s[7] = 0x0000000000000000              
s[8] = 0x0000000000000200                                  
s[9] = 0xffffffd0c8408000                                  
s[10] = 0x00000000000000c0             
s[11] = 0xffffffd0c8408010                                 
a[0] = 0x0000000000000001                                  
a[1] = 0x0000000040375b86 [rxR,0x0000000040356000-0x0000000040393000]                                                 
a[2] = 0xfffffffffffffffe [rwxRW,0x0000000000000000-0xffffffffffffffff]                                               
a[3] = 0x0000000000000000 [rwxRW,0x0000000000000000-0xffffffffffffffff]                                               
a[4] = 0x0000000040356000 [rwxRW,0x0000000040356000-0x0000000040393000]                                               
a[5] = 0x000000000001fb86                                  
a[6] = 0x0000000040356000                                  
a[7] = 0x000000000003d000               
sepc = 0xffffffc0005ac242 [rwxRW,0x0000000000000000-0xffffffffffffffff]                                                                                                                                                                      
ddc = 0x0000000000000000 [rwxRW,0x0000000000000000-0xffffffffffffffff]
sstatus == 0x0000000200000120
stval == 0x0000000000000181
panic: CHERI exception 0x1 at 0xffffffc0005ac242

cpuid = 0
time = 1654687709
KDB: stack backtrace:
db_trace_self() at db_trace_self
db_fetch_ksymtab() at db_fetch_ksymtab+0x170
kdb_backtrace() at kdb_backtrace+0x2c
vpanic() at vpanic+0x154
panic() at panic+0x52
do_trap_supervisor() at do_trap_supervisor+0x13e
cpu_exception_handler_supervisor() at cpu_exception_handler_supervisor+0xea
--- exception 28, tval = 0x181
vmspace_swap_count() at vmspace_swap_count+0xc6a
bufdone() at bufdone+0x4c
vmspace_swap_count() at vmspace_swap_count+0x1752
biodone() at biodone+0x120
g_io_deliver() at g_io_deliver+0x1e0
g_std_done() at g_std_done+0x5c
biodone() at biodone+0x120
g_io_deliver() at g_io_deliver+0x1e0
g_std_done() at g_std_done+0x5c
biodone() at biodone+0x120
g_io_deliver() at g_io_deliver+0x1e0
disk_resize() at disk_resize+0xd02
biodone() at biodone+0x120
vtmmio_attach() at vtmmio_attach+0x9686
virtqueue_intr() at virtqueue_intr+0xa
vtmmio_attach() at vtmmio_attach+0x2a80
db_dump_intr_event() at db_dump_intr_event+0x730
fork_exit() at fork_exit+0x68
fork_trampoline() at fork_trampoline+0xa
KDB: enter: panic
[ thread pid 12 tid 100027 ]
Stopped at      kdb_enter+0x4c: sd      zero,0(a0)

[bsdjhb]

So from stval it is a cause of 1 (length violation) and register index of 12 (c12 / ca2).

[bsdjhb]

I think the reason is that you can't seal a sentry cap with CSeal but instead need to use CSealEntry as the otype for a sentry exceeds cap_max_otype.

[nwf]

OK, after #1406 is applied, we're left with a pretty clearly a case of corruption leading to us attempting to reconstruct a capability we shouldn't be reconstructing. I've rebuilt my kernel in the interim (sorry), so the code leading up to the panic is now

ffffffc0005ae6bc: 0f a6 0a ff   lc      ca2, -16(s5)
;       base = cheri_getbase(cap);
ffffffc0005ae6c0: db 08 26 fe   cgetbase        a7, ca2
;       len = cheri_getlen(cap);
ffffffc0005ae6c4: db 02 36 fe   cgetlen t0, ca2
;       newcap = swap_restore_cap;
ffffffc0005ae6c8: 0f 28 09 00   lc      ca6, 0(s2)
;       offset = cheri_getoffset(cap);
ffffffc0005ae6cc: db 07 66 fe   cgetoffset      a5, ca2
;       perm = cheri_getperm(cap);
ffffffc0005ae6d0: db 06 06 fe   cgetperm        a3, ca2
;       sealed = cheri_getsealed(cap);
ffffffc0005ae6d4: 5b 07 56 fe   cgetsealed      a4, ca2
;       newcap = cheri_setoffset(newcap, base);
ffffffc0005ae6d8: db 05 18 1f   csetoffset      ca1, ca6, a7
;       newcap = cheri_setbounds(newcap, len);
ffffffc0005ae6dc: db 85 55 10   csetbounds      ca1, ca1, t0

and the register dump says

s[5] = 0xffffffd03901f400
a[1] = 0x5700000000000000 [rwxRW,0x0000000000000000-0xffffffffffffffff]
a[2] = 0x2800363332002074 [W,0x5700000000000000-0x56ffffffffffffff] (invalid,sealed)
a[3] = 0x0000000000018220
a[4] = 0x0000000000000001
a[5] = 0xd100363332002074
a[6] = 0x0000000000000000 [rwxRW,0x0000000000000000-0xffffffffffffffff]
a[7] = 0x5700000000000000
t[0] = 0xffffffffffffffff

and ddb can give us the raw memory around s5:

db> x/x 0xffffffd03901f3e0,16
0xffffffd03901f3e0:     69766168        a72756f         49000a00        7265736e
0xffffffd03901f3f0:     32002074        28003633        20746572        32203d3d
0xffffffd03901f400:     7c7c2029        74702820        3d3d2072        69726f20
0xffffffd03901f410:     46002967        656c6961        6f742064        74656720
0xffffffd03901f420:     73797320        206d6574        646e6172        656e6d6f
0xffffffd03901f430:     46007373        656c6961

Unfortunately for us, the "capability" in question is at 0xffffffd03901f3f0, which, if we ask ddb a little differently...

db> x/bc 0xffffffd03901f3f0,16
0xffffffd03901f3f0:     t \000236\000(ret == 2) || (

is almost surely not meant to be a capability.

I think #1406 should land as it clearly is a fix, but it's not the fix.

[brooksdavis]

I wonder if it would be worth taking the time to add an HMAC for swapped pages. It's something we likely want anyway in a production CHERI system.