Add library-only mode

[ayrtonm]

To support arbitrary control-flow between compartments we need to rewrite all sources (at least those at the compartment boundaries) at once. This let's us handle cases that are less likely like direct calls from a library to the main binary or calls between unrelated libraries. While this may be expected for plugin-style architectures, it is more functionality than we'll probably need and rewriting all source files at once might not be practical for deployment.

To avoid this issue we should support a library-only mode that allows you to compartmentalize libraries one at a time in an application-agnostic way. We can do this by adding these restrictions:

• libraries in different compartments can't call each other
• libraries can't call the main binary directly

Since direct calls only go from the main binary to libraries we can generate those call gates by just rewriting each library one at a time. For indirect calls we need to handle these cases. Let's say the binary is A, the lib is B and that pointers are in shared memory and can be called from either side.

Indirect cases:

• callsite in binary:
  • raw ptr from A: no callsite wrapper, no wrapper on ptr, indirect call stays in A
  • wrapped ptr from B: no callsite wrapper, ptr wrapper goes from A -> B
  • callsite in lib:
  • raw ptr from A: callsite wrapper goes from B -> A, no ptr wrapper, indirect call goes from B -> A
  • wrapper ptr from B: callsite wrapper goes from B -> A, ptr wrapper goes from A -> B, call goes from B -> A -> B

We mainly assign libraries pkeys so they can't mess with each other, so let's assume that all sensitive data is in the main binary. The issue is then preventing either callsite from calling a library function with pkey A.

We know all library functions that may be called indirectly at build-time so I think we can do this by making sure that their wrappers are the only things in the library with endbr. We can use -mmanual-endbr -fcf-protection=full and insert __attribute__((cf_check)) on address-taken function definitions in the library for this. Then only valid pointers in the library can be called indirectly since they'll be the only ones starting with endbr. @rinon thoughts?

[ayrtonm]

Not sure how useful this is, but one idea to verify the extra assumptions made by library-only mode about allowed control-flow is to unconditionally assert the PKRU before calling the target function.

[ayrtonm]

Another thing to keep in mind is that the way we currently assign compartments pkeys isn't as flexible as the rewriter makes it seem. __wrap_main in libia2.a already assumes the main compartment has pkey 1 (it's not super well-documented or enforced/validated anywhere else yet) so fixing the main compartment pkey wouldn't be a new restriction.

[ayrtonm]

I think this is probably blocked by #326.