RFC: Garbage Collection

[sempervictus]

Over the years of watching Meterpreters grow and die, i've noticed that we generally get more traction in the higher-level language implementations in terms of "commiter diversity," if that's a viable measure. One of the major difficulties i've seen higher-level language programmers have when dealing with C is memory management - figuring out a safe place to destroy something can be frustrating and cause them to simply leave the allocation unattended. Often times in their first forays this isn't even a consideration and bad things happen after a bunch of code is written since exhaustion-type failures are cumulative in nature.

Concurrently, i've observed memory leaks in mettle from several iterations ago which actually caused a pretty unpleasant crash in the control plane of a switch i'd rigged with a crontab execution of a mettle binary (what? it works...). While it didn't kill the ASIC or impede dataplane operations, the control plane went resource starved and the whole thing needed a reload. We generally dont want this to happen. If we were to implement GC as a core function of the mettle payload, it may resolve time spent in hunting this stuff. Which reminds me - we need documentation on methodologies for performing common debugging tasks against mettle payloads (including building without debug mode for things like this because memory leaks can happen differently in debug/clean builds).

The first thing that comes to mind as a potential approach is http://www.hboehm.info/gc, which is a tried and true "industry standard" C/C++ GC used in many current codebases. My understanding is that it can be built against MUSL as Gentoo seems to have an ebuild with minor patches for this. Could come in quite handy when we try to address the problem of unloading extensions and freeing up the resulting memory. Obviously resulting "weight" would be a consideration, but if it adds a couple K to the final payload, it may well be worth while.

Thoughts?

[maxvonhippel]

If I understand correctly, you raise 2 points. 1. People don't know where to garbage collect. 2. People don't garbage collect. Regarding the latter - it would be easy enough to say that any pull request must run with Valgrind and get the same or less leakage as the current HEAD it aspires to merge with. And enforcing this relatively simple policy would also lead contributors to actually bother to learn good garbage collection.

[busterb]

There is actually enough functionality in mettle (namely sigar), that we can also set some upper bounds on memory usage as well, and either make the payload kill itself or internally misbehaving things. Also the extension mechanism I have in mind should be able to enforce some limits on resource consumption so good/bad/ugly code can all coexist without stomping on each other or the target.

[busterb]

I might suggest we also add some way to do remote monitoring too. The logging subsystem is designed to log to an internal ringbuffer, so we could probably also send self stats. In constrained/embedded environments, this can sometimes be the only way to know what your resource utilization is.

[busterb]

We could probably also add something like periodic malloc_trim() calls to make sure that we are really freeing objects back to the OS aggressively, since most modern libc allocators are more optimized toward machines with hundreds of MB of ram :)

[enty8080]

I am not sure if it memory leak or something normal, but calling screenshare (https://github.com/rapid7/mettle/blob/a2f3c756ea84f07b80d64ff815cfec2b2e6d27e3/mettle/src/stdapi/ui/osx_desktop.m#L6) causes an enormous memory usage and by enormous I mean that it can reach few good GBs if not interrupted.