fortanix / rust-sgx

The Fortanix Rust Enclave Development Platform
https://edp.fortanix.com
Mozilla Public License 2.0
429 stars 98 forks source link

Add record keeping for debugging #628

Open Goirad opened 1 month ago

Goirad commented 1 month ago

This PR has the enclave runner keep track about all of the usercalls that enclaves do over their lifetimes. This includes:

The intended integration path is for consumers of this library to set up signal handling or some other mechanism to trigger collecting and logging the stats. The included formatting looks like this (from a web server enclave using async usercalls):

Async usercall counts: read: 41, write: 33, accept_stream: 3, connect_stream: 11, insecure_time: 9
Sync usercall count mappings:
Address: 0x00007fc08e2af000
    Sync Totals: launch_thread: 1, wait: 80, send: 77
    Wait Totals: RETURNQ_NOT_EMPTY: WAIT_NO: 0 WAIT_INDEFINITE: 79 OTHER: 0
                 UNPARK: WAIT_NO: 0 WAIT_INDEFINITE: 1 OTHER: 0
    Send Totals: UNPARK: 77
Address: 0x00007fc08b276000
    Sync Totals: bind_stream: 3, launch_thread: 1, wait: 40, send: 2, alloc: 12, free: 12
    Wait Totals: UNPARK: WAIT_NO: 0 WAIT_INDEFINITE: 7 OTHER: 33
    Send Totals: UNPARK: 2
Address: 0x00007fc08823d000
    Sync Totals: launch_thread: 1, wait: 3, send: 2, alloc: 5
    Wait Totals: UNPARK: WAIT_NO: 0 WAIT_INDEFINITE: 3 OTHER: 0
    Send Totals: UNPARK: 2
Address: 0x00007fc08722a000
    Sync Totals: write: 6, close: 2, launch_thread: 8, wait: 13, send: 6, alloc: 19, free: 23, async_queues: 1
    Wait Totals: UNPARK: WAIT_NO: 0 WAIT_INDEFINITE: 13 OTHER: 0
    Send Totals: UNPARK: 6
Address: 0x00007fc08d29c000
    Sync Totals: close: 8, launch_thread: 1, wait: 36, send: 5, alloc: 40, free: 40
    Wait Totals: UNPARK: WAIT_NO: 0 WAIT_INDEFINITE: 36 OTHER: 0
    Send Totals: UNPARK: 5

This is just an initial pass, I definitely need to add more robust and/or elegant logic related to the absolute number of usercalls and stringifying them.

Goirad commented 1 month ago

Instead of aggregating everything already, wouldn't it be better to have some strace-style logging of every call and then do post-processing later?

That seems a lot harder, because the application's stdout/stderr is intermingled with the enclave runner's stdout/stderr. Also, the overhead of printing that is non-trivial, and would probably have to be opted into, while this atomic based aggregation is essentially zero-cost.