Create instructions/scripts for building libc's and other things using the allocator

[sempervictus]

Since this is core infrastructure at the software tier, wide application will require wide testing. Suggest creating scaffold scripts for building libc and such with this (pkgbuilds?) in order to build test and produce binaries which can be run-tested. I'd like to try and rebuild all of arch userspace against this starting with libc.

[thestinger]

This commit integrates it into Android's Bionic libc as the malloc implementation for all 64-bit processes on Android:

https://github.com/AndroidHardening/platform_bionic/commit/04a73125db3a67f74337f7baa1dd8cd957cd062c

That's the entirety of the integration work for Android outside of this repository. The rest of the commits related to it in the Android source tree will be fixing latent memory corruption bugs in the upstream code or other compatibility issues, which should be minimal.

It's not compatible with using RLIMIT_AS as a poor way of limiting memory usage since it limits mapped address space as a whole rather than accountable mappings so usage of that needs to be worked around. The same thing applies to mitigations based on shadow regions along with debugging tools like ASan, TSan, MSan, etc. so it's not a unique compatibility issue. Limiting actual memory usage with memory control groups is generally a better option and can take into account kernel memory usage, unlike the poor approximation of using RLIMIT_AS.

I don't plan on developing or maintaining patches for integrating it into glibc myself, but it implements the glibc malloc API and that should be straightforward. It's meant to be properly integrated into libc alongside other hardening including more isolated memory regions. Both glibc and musl also need some other infrastructure developed like cross-DSO CFI integration and hardening they're missing that's present in the existing Bionic.

This can already be tested with glibc and musl by dynamically linking it or using LD_PRELOAD. Since musl has proper static linking support and aims to support robust replacement of the malloc implementation, it should also be possible to use it as a replacement for the standard malloc in a statically linked musl-based executable but that's untested and may require some changes. It's not meant to be used this way in production because it's intended to go along with other important libc hardening including supporting features like cross-DSO Clang CFI like Bionic and additional features. Some of those features are closely related and you can look at my past Bionic libc hardening work to see what I mean.

[sempervictus]

Thanks for the detailed explanation. When/if you get a chance, could you link some of the relevant prior commits? "Looking at your past work" is sort of akin to "skimming the Encyclopedia Britannica."

[thestinger]

One of the related features is implementing an isolated region for dynamic libraries, including the ones loaded at initialization and those loaded via dlopen. That's similar to the isolated regions for different size classes in this allocator and fits well alongside it.

There was also an implementation of dynamic overflow checks for system calls via the malloc_object_size feature added to the extended port of OpenBSD malloc and a dynamic_object_size wrapper around it adding more features around it like fast paths for objects in global data or on the current stack. I haven't done a very thorough implementation of malloc_object_size for this allocator yet. It's just a working stub that's not nearly as good as it could be, since it will be possible to return fully accurate results.

The libc also needs an implementation of cross-DSO CFI support and cross-DSO SafeStack support for those features to work properly. See https://struct.github.io/cross_dso_cfi.html about one of those features. It should also have proper setjmp / longjmp hardening like Bionic (better than in glibc), protection for all the internal function pointers (which is something I worked on greatly improving out-of-tree), etc. It's just not very reasonable to build on glibc when it's so overly complicated with lots of complex correctness and security flaws in the design.

[sempervictus]

While Android and other Google supported or sourced things use clang, most POSIX systems are still on GCC, and most Linux' use glibc. CFI, in that world, is a mess given that the best implementation is documented like an AK103 for English speakers, and effectively nonpublic due to trademarks or copyrights or something to do with IP (not that the AK creators nation is going to not use or not improve on them based on the restrictions which keep democratically inclined developers from doing so). That said, IMHO, CFI as a target defense semantic is actually about as accurate as saying "anti-air" which is anything from altimeter induced detonation fragmentation rounds to guided munitions propelled and oriented by their own faculties. @thestinger likely knows better than almost anyone that even in the same CFI implementation, the devil is very much in the details (clang CFI being brutal to all comers by introducing a 2nd level of indirection and compile flags which result in drastic coverage reductions). In that vein, yeah, CFI is the brass ring of defenses for memory corruption presuming perfect 1:1 mappings of call and return sites and that we don't arbitrarily write everything our shellcode needs in order to work (a microkernel for attackers) into memory... Clang and llvm in general are the future, what we think of as plugins today for GCC are pipeline components there (kind of, respectre for instance is a good example of this, rap is not) and "all the cool kids are doing it" (Daniel Micay is a cool kid, for the actual record). That said, it can't be adopted world-wide while the current sociology of Linux persists - they think (free > safe && free > correct||safe && free == gplv2). There are discussions afoot between people who care and those in control, but the general response we get is "if you front a few million bucks to get this all production ready, we may consider not rejecting you outright." ... it's a long road to clang as default if even "semantically correct C" is a problem (constify and size overflow plugins make corrections to c, stackleak fixes compiler generated assembly to make what c should produce, and that was outright rejected). Is there a way to have the build system validate libc or other dependency options and functions at compile-time? Kmods do this all the time (SPL, ZFS, SCST, PF_RING, etc)? If so, then maybe the "nice to have" pieces get ifdeffed to constants permitting the function and have safe paths for the real world of today which may not have size-based-xory-func?

[thestinger]

Is there a way to have the build system validate libc or other dependency options and functions at compile-time? Kmods do this all the time (SPL, ZFS, SCST, PF_RING, etc)? If so, then maybe the "nice to have" pieces get ifdeffed to constants permitting the function and have safe paths for the real world of today which may not have size-based-xory-func?

This project is already compatible with musl and glibc. You can build it as a standalone dynamic library with the Makefile and test it with them using LD_PRELOAD. That's the only purpose of the Makefile since Android uses the declarative Android.bp build system (https://github.com/AndroidHardening/hardened_malloc/blob/master/Android.bp) to build this rather than make. Proper integration in the libc can't be done in this repository since it involves forking the libc implementation and replacing the malloc implementation with this one internally as I did for Bionic with https://github.com/AndroidHardening/platform_bionic/commit/04a73125db3a67f74337f7baa1dd8cd957cd062c.

However, as I've mentioned, there's a lot more to making a hardened libc than including this allocator. A hardened variant of musl would be a separate project from this requiring a developer with funding. I don't think doing that for glibc is feasible as it's too much of a mess with far too much complexity and legacy cruft to grapple with it. Forking musl and replacing the malloc implementation with this would be a good start but there's a lot more to do like setjmp/longjmp protection (ideally on par with Bionic, not the inferior glibc implementation), internal function pointer protection, cross-DSO CFI and SafeStack support, an isolated library region, secondary stack randomization / better guard pages, a modern _FORTIFY_SOURCE implementation (unlike glibc where it only covers writes, not reads), etc.

free == gplv2

Permissive licenses like MIT, BSD and Apache 2 are more free than GPL2 and especially GPL3 no matter how much people try to spin it. It's also not capable of making a project sustainable or getting people to contribute back changes. GCC and glibc even require copyright assignment for all non-trivial contributions so that drives away most contributions and outright prevents the fantasy of the GPL resulting in code being contributed back when it wouldn't have been otherwise.

[thestinger]

I've done everything that I can do to support glibc and musl in this repository and I regularly test with both so I'm going to close this issue as completed in terms of what's in scope for this repository. Proper integrating into a libc requires a fork replacing the malloc implementation, and I think it only makes sense as part of a broader project hardening the libc implementation as I intend to do for Bionic again. Bionic is a better starting point even without that though, since Google has done a lot of work on security and also many of my smaller changes were merged.

I would like to be involved in making a hardened variant of musl, but I can't do more than guiding development. Other developers are going to need to step up and do some work, rather than leaving it all the Google or expecting me to do it. I find it quite sad that CFI and SafeStack have no production / security-oriented implementations for traditional Linux distributions and are not being adopted. Google is doing nearly all the security work in the ecosystem and Android/ChromeOS are the primary adopters of it. HardenedBSD has adopted CFI and SafeStack in addition to various other mitigations though, and other distributions should follow their example rather than sticking with 2004 era mitigations that are still not even consistently enabled by them.

[sempervictus]

Thanks for the detailed explanations, as always. For the MUSL hardening work - do you have enough contact with them to determine if such an effort would be upstreamed? We can discuss scope and cost offline, but if your work can find its way into a widely adopted ABI, we would be interested in funding some or all of that effort - there needs to be a usable gold standard before users can even start to consider the difference.