Create tool to validate instrumented binaries

[ayrtonm]

When dealing with code generated from templates it's easy to miss source files which need to be compiled with certain flags to make sure cross-compartment direct calls go through symbol versioning wrappers.

It might be a good idea to create a small script to verify that the final binaries are instrumented as intended. For direct calls this should definitely be doable by either parsing readelf/nm output or with pyelftools or similar. For indirect calls I'd have to think a little bit about what we can easily verify from just the binaries without disassembly.

[thedataking]

This sounds like a good idea and possibly something I can pick up.

For direct calls this should definitely be doable by either parsing readelf/nm output or with pyelftools or similar.

Any thoughts on the drawbacks of doing this dynamically vs statically (other than the usual trade offs)? I think it would be quite easy to write a pin tool that monitors compartment transitions dynamically as they happen. It is maintained by Intel and fairly easy to use in my experience.

[ayrtonm]

Besides the usual coverage issue, I don't see any issue with doing this dynamically. I just think it'd be easier to do statically, but if we can handle both direct and indirect calls by instrumenting binaries with the same tool that could be the better choice.

monitors compartment transitions dynamically as they happen

I think we're probably on the same page, but just to clarify we'd want to monitor cross-DSO calls (ignoring calls to dependencies #133) in case we forgot to add a compartment transition somewhere.

[thedataking]

Note to self: see compartment runtime initialization for the the PKRU properties that the tool should be validating.

[ayrtonm]

@endbr64 pointed out some details from debugging nginx that are relevant to the issue in the first comment. The code generated from the template is located in nginx.so which is separate from both the main nginx executable and the perl module ngx_http_perl_module.so. In this case the perl module calls into nginx.so which calls into the main nginx executable, though we had previously assumed that cross-compartment direct calls only go from the main executable to the compartmentalized module and vice-versa.

This means that my vague, initial idea (run nm/readelf --dyn-syms on the executable and module, then parse the list of undefined function names and compare with the function wrappers) would have likely missed the issue since the functions that weren't wrapped were in a third shared object (nginx.so). A static tool using ldd could've caught this, but I haven't fully thought through this.

It's more likely that a dynamic tool would've caught the issue, though handling this specific case (i.e. treating multiple shared objects as a single compartment) may be more complicated. I think there's still value in a dynamic tool that doesn't handle this specific case since the issue was two-fold: (1) the source code was hidden underneath a few layers of build system glue and code generation and (2) one compartment made calls to the other from two shared objects which broke our assumptions. @thedataking we should start off with something simple, but I thought you should be aware of this.

[thedataking]

Notes from initial exploration of pin (pin-3.24-98612-g6bd5931f2-gcc-linux).

Seeing problem with simple1-main binary on simple built-in pin-tool (on both AMD Zen3 and Intel Gen 12 CPUs):

../../../pin -t obj-intel64/opcodemix.so -- path/to/build/header-rewriter/tests/simple1/simple1-main
E: PIN_RaiseException() raised invalid exception: RECEIVED_UNKNOWN.

On the other hand, the minimal-main binary seems to work fine (tested on both AMD and Intel).

[ayrtonm]

I think this is either caused by our indirect function wrappers or the signal handler for our tests (see include/test_fault_handler.h). You should run ./header-rewriter/tests/untrusted_indirect/untrusted_indirect-main foo to see if function pointer wrappers work and ./header-rewriter/tests/should_segfault/should_segfault-main to see if the signal handler works.

Also I'm assuming this was before we switched from symbol versioning to ld --wrap for inserting call gates in #149. You can see if that makes a difference by rerunning on main though I doubt it since we insert call gates the same way for all tests.