Open zhuyifei1999 opened 1 year ago
I rewrote the logic in C so it'll hopefully be more clear https://github.com/zhuyifei1999/zapps-poc/blob/master/zapps-crt0.c
Awesome.
I agree the shim process part of Zapps is potentially extraneous, and a somewhat unattractive hack.
Much of what we've shipped and called Zapps so far is using a very placeholder, minimum-viable-product implementation of the shim, as well. It's big, bulky, and certainly can be improved -- even within the category of shims that are doing multiple exec's.
And this goes somewhere even further than the multiple-exec space. Super awesome. I had some idea that something like this should be possible, but it's completely spectacular to see it fully worked, and even proven by working demo.
So I guess most questions that occur to me will be about what kinda of interfaces/ABI/behavioral-contracts this becomes sensitive to, and what that will imply about portability/maintainability/fragility.
A few questions and scattered first impressions, in no particular order:
One fewer exec is obviously a flat out win. No further comments on that. ^^
I see in the c version, you calmly reimplement a good chunk of a minimal libc. (I completely understand why, especially compared to glibc.)
Elf64_auxv_t
. Do we need to own that? Is there a way we could avoid it?.S
file again later, as well as the .c
; it may contain the answer to my questions already and I just haven't mentally unpacked it yet. I certainly notice the .S
is quite short!)What're the boundaries of portability for this? For example I see a lot of small details which are setting things up with attention to what glibc and the matching gnu ELF interp require... Is all this going to work transparently equally well if we try to deploy it rolled together with a program compiled against musl or some other libc, without modifications?
The patching away of the problematic glibc assert at runtime is neat and evidently blasts its way through a problem, but also seems kinda scary. I'm worried that changing executable pages of memory at runtime is going to trip virus detection heuristics or other security boundaries that may exist in some contexts.
I look forward to looking at this in greater depth soon!
I see in the c version, you calmly reimplement a good chunk of a minimal libc. (I completely understand why, especially compared to glibc.)
I think a lot of the functions are of two kinds - string functions and syscall functions.
_zapps_syscall
directly with the SYS_*
numbers instead of using the wrappers. I added wrappers so it's more clear what the arguments are.__builtin_memcpy
. Unfortunately when I was experimenting with built-ins I believe strrchr
's built-in emitted a call to strrchr's PLT entry. Since dynamic linking don't work at that point it immediately segfaulted and I decided to ignore built-ins for the time being.Especially the parts unpacking e.g. Elf64_auxv_t. Do we need to own that? Is there a way we could avoid it?
Do you mean _zapps_getauxval_ptr
? I'm not sure there's a way to guarantee the position of each auxv in the auxv array.
As for the part where it skips argc
argv
envp
, it's required to find auxv itself. The logic is very silmilar to both glibc and musl:
And we need to patch auxv to provide the _start
symbol to ld.so via AT_ENTRY
vector, even if we ignore how I patched AT_BASE too.
I certainly notice the .S is quite short!
It's basically the same code heh, slightly inferior. The commit message (https://github.com/zhuyifei1999/zapps-poc/commit/916205660971ee42c54a7106730a01e7e802c7dd) describes what changed in functionality in the rewrite.
What're the boundaries of portability for this? For example I see a lot of small details which are setting things up with attention to what glibc and the matching gnu ELF interp require... Is all this going to work transparently equally well if we try to deploy it rolled together with a program compiled against musl or some other libc, without modifications?
I don't see why it would cause an issue. That patch to PT_INTERP
is to revert it back from PT_ZAPPS_INTERP
(which is done by strip_interp.c). My main concern for something breaks is if a libc cares about the not just the name of ld.so
, but also the path of it, from reading PT_INTERP segment. I think the most likely candidate for a libc that would care is glibc, but if it doesn't I don't find it likely that other libcs would care at all.
That said, I haven't tested with musl or other libcs yet so I'm not 100% sure.
The patching away of the problematic glibc assert at runtime is neat and evidently blasts its way through a problem, but also seems kinda scary. I'm worried that changing executable pages of memory at runtime is going to trip virus detection heuristics or other security boundaries that may exist in some contexts.
In the C version I used a pwrite on /proc/self/mem instead, which seems like a much better way to do this (https://offlinemark.com/2021/05/12/an-obscure-quirk-of-proc/). No more RWX ;)
Wait I misread
The patching away of the problematic glibc assert at runtime
No I didn't do this. The patch was to revert PT_INTERP back from PT_ZAPPS_INTERP. This doesn't modify any glibc code, but only the segmen headers of the ELF we are loading. The X bit was set because certain older gccs, .text starts within the first mapped page.
I use the PT_ZAPPS_INTERP
patch so the kernel invokes the executable directly without an interpreter.
That said, I haven't tested with musl or other libcs yet so I'm not 100% sure.
I tested musl with a cross-compiler. Interestingly in musl ld.so symlinks to libc.so:
zhuyifei1999@zhuyifei1999-ThinkPad-P14s-Gen-2a ~/zapps-poc $ ls -l /usr/x86_64-pc-linux-musl/lib/ld-musl-x86_64.so.1
lrwxrwxrwx 1 root root 41 Dec 30 18:43 /usr/x86_64-pc-linux-musl/lib/ld-musl-x86_64.so.1 -> /usr/x86_64-pc-linux-musl/usr/lib/libc.so
zhuyifei1999@zhuyifei1999-ThinkPad-P14s-Gen-2a ~/zapps-poc $ ls -l /usr/x86_64-pc-linux-musl/usr/lib/libc.so
-rwxr-xr-x 1 root root 817656 Dec 30 18:43 /usr/x86_64-pc-linux-musl/usr/lib/libc.so
And it defines PAGE_SIZE
and lacks <error.h>
, so with that in mind, I did https://github.com/zhuyifei1999/zapps-poc/commit/48bb4c675ee5faeaedecbbd6b316b1b3bcd003ff
And it totally just works:
zhuyifei1999@zhuyifei1999-ThinkPad-P14s-Gen-2a ~/zapps-poc $ make CC=x86_64-pc-linux-musl-gcc
mkdir -p absolute
x86_64-pc-linux-musl-gcc -o absolute/lib.so lib.c -fPIC -shared -g -Os -pipe
x86_64-pc-linux-musl-gcc -o absolute/exe exe.c -L absolute -l:lib.so -Wl,-rpath=absolute -g -Os -pipe
mkdir -p tmp
x86_64-pc-linux-musl-gcc -o tmp/strip_interp strip_interp.c -g -Os -pipe
x86_64-pc-linux-musl-gcc -o tmp/zapps-crt0.o zapps-crt0.c -fPIC -ffreestanding -fno-merge-constants -c -g -Os -pipe
mkdir -p relative
x86_64-pc-linux-musl-gcc -o relative/lib.so lib.c -fPIC -shared -g -Os -pipe
cp $(x86_64-pc-linux-musl-gcc --print-file-name=libc.so) relative/libc.so
x86_64-pc-linux-musl-gcc -o relative/exe exe.c -L relative -l:lib.so -Wl,-rpath=XORIGIN -Wl,-e_zapps_start -Wl,--unique=.text.zapps tmp/zapps-crt0.o -g -Os -pipe
sed -i '0,/XORIGIN/{s/XORIGIN/$ORIGIN/}' relative/exe
relative/libc.so tmp/strip_interp relative/exe
zhuyifei1999@zhuyifei1999-ThinkPad-P14s-Gen-2a ~/zapps-poc $ relative/exe
static_constructor in lib invoked
static_constructor in exe invoked
main invoked with arguments:
argv[0] = relative/exe
foo invoked
contents of /proc/self/maps:
555556007000-555556008000 ---p 00000000 00:00 0 [heap]
555556008000-555556009000 rw-p 00000000 00:00 0 [heap]
7f349a3a8000-7f349a3a9000 r--p 00000000 00:2b 55447740 /home/zhuyifei1999/zapps-poc/relative/lib.so
7f349a3a9000-7f349a3aa000 r-xp 00001000 00:2b 55447740 /home/zhuyifei1999/zapps-poc/relative/lib.so
7f349a3aa000-7f349a3ab000 r--p 00002000 00:2b 55447740 /home/zhuyifei1999/zapps-poc/relative/lib.so
7f349a3ab000-7f349a3ac000 r--p 00002000 00:2b 55447740 /home/zhuyifei1999/zapps-poc/relative/lib.so
7f349a3ac000-7f349a3ad000 rw-p 00003000 00:2b 55447740 /home/zhuyifei1999/zapps-poc/relative/lib.so
7f349a3ad000-7f349a3c2000 r--p 00000000 00:2b 55447741 /home/zhuyifei1999/zapps-poc/relative/libc.so
7f349a3c2000-7f349a43e000 r-xp 00015000 00:2b 55447741 /home/zhuyifei1999/zapps-poc/relative/libc.so
7f349a43e000-7f349a474000 r--p 00091000 00:2b 55447741 /home/zhuyifei1999/zapps-poc/relative/libc.so
7f349a474000-7f349a475000 r--p 000c6000 00:2b 55447741 /home/zhuyifei1999/zapps-poc/relative/libc.so
7f349a475000-7f349a476000 rw-p 000c7000 00:2b 55447741 /home/zhuyifei1999/zapps-poc/relative/libc.so
7f349a476000-7f349a479000 rw-p 00000000 00:00 0
7f349a479000-7f349a47a000 r--p 00000000 00:2b 55447743 /home/zhuyifei1999/zapps-poc/relative/exe
7f349a47a000-7f349a47b000 r-xp 00001000 00:2b 55447743 /home/zhuyifei1999/zapps-poc/relative/exe
7f349a47b000-7f349a47c000 r--p 00002000 00:2b 55447743 /home/zhuyifei1999/zapps-poc/relative/exe
7f349a47c000-7f349a47d000 r--p 00002000 00:2b 55447743 /home/zhuyifei1999/zapps-poc/relative/exe
7f349a47d000-7f349a47e000 rw-p 00003000 00:2b 55447743 /home/zhuyifei1999/zapps-poc/relative/exe
7ffd719e6000-7ffd71a08000 rw-p 00000000 00:00 0 [stack]
7ffd71b04000-7ffd71b08000 r--p 00000000 00:00 0 [vvar]
7ffd71b08000-7ffd71b0a000 r-xp 00000000 00:00 0 [vdso]
zhuyifei1999@zhuyifei1999-ThinkPad-P14s-Gen-2a ~/zapps-poc $ cp -r relative/ /tmp/bar
zhuyifei1999@zhuyifei1999-ThinkPad-P14s-Gen-2a ~/zapps-poc $ cd /mnt
zhuyifei1999@zhuyifei1999-ThinkPad-P14s-Gen-2a /mnt $ /tmp/bar/exe
static_constructor in lib invoked
static_constructor in exe invoked
main invoked with arguments:
argv[0] = /tmp/bar/exe
foo invoked
contents of /proc/self/maps:
555556bdb000-555556bdc000 ---p 00000000 00:00 0 [heap]
555556bdc000-555556bdd000 rw-p 00000000 00:00 0 [heap]
7f7cce472000-7f7cce473000 r--p 00000000 00:22 31942 /tmp/bar/lib.so
7f7cce473000-7f7cce474000 r-xp 00001000 00:22 31942 /tmp/bar/lib.so
7f7cce474000-7f7cce475000 r--p 00002000 00:22 31942 /tmp/bar/lib.so
7f7cce475000-7f7cce476000 r--p 00002000 00:22 31942 /tmp/bar/lib.so
7f7cce476000-7f7cce477000 rw-p 00003000 00:22 31942 /tmp/bar/lib.so
7f7cce477000-7f7cce48c000 r--p 00000000 00:22 31943 /tmp/bar/libc.so
7f7cce48c000-7f7cce508000 r-xp 00015000 00:22 31943 /tmp/bar/libc.so
7f7cce508000-7f7cce53e000 r--p 00091000 00:22 31943 /tmp/bar/libc.so
7f7cce53e000-7f7cce53f000 r--p 000c6000 00:22 31943 /tmp/bar/libc.so
7f7cce53f000-7f7cce540000 rw-p 000c7000 00:22 31943 /tmp/bar/libc.so
7f7cce540000-7f7cce543000 rw-p 00000000 00:00 0
7f7cce543000-7f7cce544000 r--p 00000000 00:22 31944 /tmp/bar/exe
7f7cce544000-7f7cce545000 r-xp 00001000 00:22 31944 /tmp/bar/exe
7f7cce545000-7f7cce546000 r--p 00002000 00:22 31944 /tmp/bar/exe
7f7cce546000-7f7cce547000 r--p 00002000 00:22 31944 /tmp/bar/exe
7f7cce547000-7f7cce548000 rw-p 00003000 00:22 31944 /tmp/bar/exe
7ffe70e0a000-7ffe70e2c000 rw-p 00000000 00:00 0 [stack]
7ffe70eed000-7ffe70ef1000 r--p 00000000 00:00 0 [vvar]
7ffe70ef1000-7ffe70ef3000 r-xp 00000000 00:00 0 [vdso]
zhuyifei1999@zhuyifei1999-ThinkPad-P14s-Gen-2a /mnt $ cd /usr/tmp
zhuyifei1999@zhuyifei1999-ThinkPad-P14s-Gen-2a /usr/tmp $ ln -s ../../tmp/bar/exe baz
zhuyifei1999@zhuyifei1999-ThinkPad-P14s-Gen-2a /usr/tmp $ ./baz
static_constructor in lib invoked
static_constructor in exe invoked
main invoked with arguments:
argv[0] = ./baz
foo invoked
contents of /proc/self/maps:
555555f25000-555555f26000 ---p 00000000 00:00 0 [heap]
555555f26000-555555f27000 rw-p 00000000 00:00 0 [heap]
7f59bca77000-7f59bca78000 r--p 00000000 00:22 31942 /tmp/bar/lib.so
7f59bca78000-7f59bca79000 r-xp 00001000 00:22 31942 /tmp/bar/lib.so
7f59bca79000-7f59bca7a000 r--p 00002000 00:22 31942 /tmp/bar/lib.so
7f59bca7a000-7f59bca7b000 r--p 00002000 00:22 31942 /tmp/bar/lib.so
7f59bca7b000-7f59bca7c000 rw-p 00003000 00:22 31942 /tmp/bar/lib.so
7f59bca7c000-7f59bca91000 r--p 00000000 00:22 31943 /tmp/bar/libc.so
7f59bca91000-7f59bcb0d000 r-xp 00015000 00:22 31943 /tmp/bar/libc.so
7f59bcb0d000-7f59bcb43000 r--p 00091000 00:22 31943 /tmp/bar/libc.so
7f59bcb43000-7f59bcb44000 r--p 000c6000 00:22 31943 /tmp/bar/libc.so
7f59bcb44000-7f59bcb45000 rw-p 000c7000 00:22 31943 /tmp/bar/libc.so
7f59bcb45000-7f59bcb48000 rw-p 00000000 00:00 0
7f59bcb48000-7f59bcb49000 r--p 00000000 00:22 31944 /tmp/bar/exe
7f59bcb49000-7f59bcb4a000 r-xp 00001000 00:22 31944 /tmp/bar/exe
7f59bcb4a000-7f59bcb4b000 r--p 00002000 00:22 31944 /tmp/bar/exe
7f59bcb4b000-7f59bcb4c000 r--p 00002000 00:22 31944 /tmp/bar/exe
7f59bcb4c000-7f59bcb4d000 rw-p 00003000 00:22 31944 /tmp/bar/exe
7ffd64bb4000-7ffd64bd6000 rw-p 00000000 00:00 0 [stack]
7ffd64bf6000-7ffd64bfa000 r--p 00000000 00:00 0 [vvar]
7ffd64bfa000-7ffd64bfc000 r-xp 00000000 00:00 0 [vdso]
As a control group comparison, this is invoking the ld.so directly on the normal binary:
zhuyifei1999@zhuyifei1999-ThinkPad-P14s-Gen-2a ~/zapps-poc $ relative/libc.so absolute/exe
static_constructor in lib invoked
static_constructor in exe invoked
main invoked with arguments:
argv[0] = absolute/exe
foo invoked
contents of /proc/self/maps:
5555558cc000-5555558cd000 ---p 00000000 00:00 0 [heap]
5555558cd000-5555558ce000 rw-p 00000000 00:00 0 [heap]
7f2071bc0000-7f2071bc1000 r--p 00000000 00:2b 55447734 /home/zhuyifei1999/zapps-poc/absolute/lib.so
7f2071bc1000-7f2071bc2000 r-xp 00001000 00:2b 55447734 /home/zhuyifei1999/zapps-poc/absolute/lib.so
7f2071bc2000-7f2071bc3000 r--p 00002000 00:2b 55447734 /home/zhuyifei1999/zapps-poc/absolute/lib.so
7f2071bc3000-7f2071bc4000 r--p 00002000 00:2b 55447734 /home/zhuyifei1999/zapps-poc/absolute/lib.so
7f2071bc4000-7f2071bc5000 rw-p 00003000 00:2b 55447734 /home/zhuyifei1999/zapps-poc/absolute/lib.so
7f2071bc5000-7f2071bc6000 r--p 00000000 00:2b 55447735 /home/zhuyifei1999/zapps-poc/absolute/exe
7f2071bc6000-7f2071bc7000 r-xp 00001000 00:2b 55447735 /home/zhuyifei1999/zapps-poc/absolute/exe
7f2071bc7000-7f2071bc8000 r--p 00002000 00:2b 55447735 /home/zhuyifei1999/zapps-poc/absolute/exe
7f2071bc8000-7f2071bc9000 r--p 00002000 00:2b 55447735 /home/zhuyifei1999/zapps-poc/absolute/exe
7f2071bc9000-7f2071bca000 rw-p 00003000 00:2b 55447735 /home/zhuyifei1999/zapps-poc/absolute/exe
7f2071bca000-7f2071bdf000 r--p 00000000 00:2b 55447741 /home/zhuyifei1999/zapps-poc/relative/libc.so
7f2071bdf000-7f2071c5b000 r-xp 00015000 00:2b 55447741 /home/zhuyifei1999/zapps-poc/relative/libc.so
7f2071c5b000-7f2071c91000 r--p 00091000 00:2b 55447741 /home/zhuyifei1999/zapps-poc/relative/libc.so
7f2071c91000-7f2071c92000 r--p 000c6000 00:2b 55447741 /home/zhuyifei1999/zapps-poc/relative/libc.so
7f2071c92000-7f2071c93000 rw-p 000c7000 00:2b 55447741 /home/zhuyifei1999/zapps-poc/relative/libc.so
7f2071c93000-7f2071c96000 rw-p 00000000 00:00 0
7ffdc47e7000-7ffdc4809000 rw-p 00000000 00:00 0 [stack]
7ffdc4972000-7ffdc4976000 r--p 00000000 00:00 0 [vvar]
7ffdc4976000-7ffdc4978000 r-xp 00000000 00:00 0 [vdso]
And this is if musl ld.so is loaded by kernel as the interpreter:
zhuyifei1999@zhuyifei1999-ThinkPad-P14s-Gen-2a ~/zapps-poc $ patchelf --set-interpreter relative/libc.so absolute/exe
zhuyifei1999@zhuyifei1999-ThinkPad-P14s-Gen-2a ~/zapps-poc $ absolute/exe
static_constructor in lib invoked
static_constructor in exe invoked
main invoked with arguments:
argv[0] = absolute/exe
foo invoked
contents of /proc/self/maps:
561685a4e000-561685a4f000 r--p 00000000 00:2b 55447735 /home/zhuyifei1999/zapps-poc/absolute/exe
561685a4f000-561685a50000 r-xp 00001000 00:2b 55447735 /home/zhuyifei1999/zapps-poc/absolute/exe
561685a50000-561685a51000 r--p 00002000 00:2b 55447735 /home/zhuyifei1999/zapps-poc/absolute/exe
561685a51000-561685a52000 r--p 00002000 00:2b 55447735 /home/zhuyifei1999/zapps-poc/absolute/exe
561685a52000-561685a53000 rw-p 00003000 00:2b 55447735 /home/zhuyifei1999/zapps-poc/absolute/exe
561685a53000-561685a54000 rw-p 00005000 00:2b 55447735 /home/zhuyifei1999/zapps-poc/absolute/exe
561687414000-561687415000 ---p 00000000 00:00 0 [heap]
561687415000-561687416000 rw-p 00000000 00:00 0 [heap]
7fcaf43cb000-7fcaf43cc000 r--p 00000000 00:2b 55447734 /home/zhuyifei1999/zapps-poc/absolute/lib.so
7fcaf43cc000-7fcaf43cd000 r-xp 00001000 00:2b 55447734 /home/zhuyifei1999/zapps-poc/absolute/lib.so
7fcaf43cd000-7fcaf43ce000 r--p 00002000 00:2b 55447734 /home/zhuyifei1999/zapps-poc/absolute/lib.so
7fcaf43ce000-7fcaf43cf000 r--p 00002000 00:2b 55447734 /home/zhuyifei1999/zapps-poc/absolute/lib.so
7fcaf43cf000-7fcaf43d0000 rw-p 00003000 00:2b 55447734 /home/zhuyifei1999/zapps-poc/absolute/lib.so
7fcaf43d0000-7fcaf43e5000 r--p 00000000 00:2b 55447741 /home/zhuyifei1999/zapps-poc/relative/libc.so
7fcaf43e5000-7fcaf4461000 r-xp 00015000 00:2b 55447741 /home/zhuyifei1999/zapps-poc/relative/libc.so
7fcaf4461000-7fcaf4497000 r--p 00091000 00:2b 55447741 /home/zhuyifei1999/zapps-poc/relative/libc.so
7fcaf4497000-7fcaf4498000 r--p 000c6000 00:2b 55447741 /home/zhuyifei1999/zapps-poc/relative/libc.so
7fcaf4498000-7fcaf4499000 rw-p 000c7000 00:2b 55447741 /home/zhuyifei1999/zapps-poc/relative/libc.so
7fcaf4499000-7fcaf449c000 rw-p 00000000 00:00 0
7ffe6d728000-7ffe6d74a000 rw-p 00000000 00:00 0 [stack]
7ffe6d754000-7ffe6d758000 r--p 00000000 00:00 0 [vvar]
7ffe6d758000-7ffe6d75a000 r-xp 00000000 00:00 0 [vdso]
So yeah, I think this method is pretty libc-agnostic :) other than the file name of the ld.so needs to be changed.
I also took a look at musl source code to see if the path of ld.so is a concern (and musl source is much easier to read than glibc), since earlier I said
I think the most likely candidate for a libc that would care is glibc, but if it doesn't I don't find it likely that other libcs would care at all.
PT_INTERP
segment affects the ldso.name
variable:
https://elixir.bootlin.com/musl/latest/source/ldso/dynlink.c#L1783/etc/ld-musl-arch.path
is relative to it: https://elixir.bootlin.com/musl/latest/source/ldso/dynlink.c#L1066Do you have any thoughts on what the next steps would be, and how we should integrate this?
I'm thinking we may want to maintain both the older boring'er way for a while, and try this in parallel. (I'm very conservative in some regards.) I don't have strong opinions about how we implement this, but if you have ideas I'm receptive.
By the way, I see you published your new code under an MIT license, and I'm happy for any open-source license like that. But just as a heads up, we tend to do Apache2-OR-MIT for other stuff, including our existing code in this repo. (This is largely guided by the licensing policies of Protocol Labs, which affects several of us by default, and is also very open-source oriented and generally pretty pleasing.) I don't think it's any problem to have code that's MIT-only floating around, but if it's okay by you to also make it Apache2-OR-MIT, it might create moderately less kerfuffle if we decide to combine these works all into one repo.
but if it's okay by you to also make it Apache2-OR-MIT
Sure
Done. https://github.com/zhuyifei1999/zapps-poc/commit/3fdfc382a053db88c72346e01cde37d97ed18ec8
Do you have any thoughts on what the next steps would be, and how we should integrate this?
I'm not familiar with how warptools does the building & packaging, so I don't think I'm the right person to ask about this. What I did was more of just a proof that the extra exec need not exist and what procedures are necessary to get rid of it.
That said, I don't see it as any more complex than building a zapps-crt0.o in some known location and for most projects add an LDFLAGS to use it.
I have another dumb question that has come to my mind :)
With this shimless crt0 approach: if we wanted to add in some further preamble code to manipulate the environment variables the process sees, what options do we have? Will it be possible to do so with consistency?
We haven't done this in the ldshim repo as it stands yet either, to be clear. But it's looking like something that might slip into our scope in the future.
For context on why: tl;dr: it seems practical. (Perhaps a bit kludgey, also. But practical.) It seems that, in addition to our linker tricks to make the initial library loading work correctly and path-agnostically, there's a good number of programs out in the wild that that will also require some additional kicking in the shins to behave path-agnostically at runtime. And sometimes environment variables we set at launch are going to be the easiest (read: not patching at build time) way to do this.
(We've started to see a few of these already. Also, Joey Hess (who has apparently been barking up the same tree as zapps for quite a while!) pointed out to us that in his experience working in a similar direction, there were lots of environment variables he found as the most sensible way to request fully path-agnostic behaviors at runtime. Examples he mentioned included LOCPATH, GCONV_PATH, GIT_TEMPLATE_DIR, MANPATH... but of course the individual cases aren't the point; more that it seems to come up fairly often in practice.)
(I'll also say: I still don't love environment variables as "the" way to solve these problems -- the inheritability of env vars in particular just doesn't feel correct or clean at all in these scenarios. But it's looking like something we should have a place for in our bag of tricks.)
With this shimless crt0 approach: if we wanted to add in some further preamble code to manipulate the environment variables the process sees, what options do we have? Will it be possible to do so with consistency?
I mean, the variables are on stack, and the stack has a very well defined layout (https://lwn.net/Articles/631631/). You can do whatever you want to, adding env vars, modifying them, or deleting them, before passing the control over to the dynamic loader.
To be honest I also think env variables are wrong. If bash is a zapp, and someone uses it to call a non-zapp program the non-zapp program should not inherit bash's workarounds. Similarly if strace is a zapp and someone passes a non-zapp for strace to invoke, that non-zapp shouldn't inherit strace zapp's workarounds.
I was reading https://zapps.app/technology/ yesterday and it ocurred to me that the shim process seemed extraneous.
ld.so
is a ELF dynamic object, so is a dynamically linked executable:Both are able to run, because there's an entry point:
But
ld.so
does not have an INTERP ELF segment:This means, to start the execution of a dynamically linked object, the INTERP segment is optional as far as the kernel is concerned. If the object has no INTERP, the kernel will load it at a randomized address and jump to the entry point.
... which got me thinking, if the entry point is reachable, we can do in userspace the what used to be the kernel's job of loading
ld.so
into the address space, right? All we need to do is fix up the auxiliary vector to makeld.so
believe nothing is out of the ordinary.So here is the POC:
I chose to write it in assembly because I was too lazy to mess with compiler options. I just want something that will work regardless of compiler. Rewriting most of it in C is on my TODO. Only the initial stage of the entry point and then the jump to
ld.so
have to be assembly, but the rest should be convertible to C.It also turned out that libc will require an INTERP segment when it's called (otherwise this assertion will occur: https://elixir.bootlin.com/glibc/glibc-2.36.9000/source/elf/rtld.c#L1291) so I patched it in at runtime. This unforunately meant that I have that page as RWX. I can re-mprotect it with the initial permissions but it needs a bit more code to find the right segment for the right permission bits.
Why do we care? I think one of the potential use cases where a jumploader could fail is in the case of binfmt-misc, or even setuids (though I'm not sure about the security of my approach either yet). In the case of binfmt-misc, the kernel would pass information about what's being executed to the binary via auxiliary vector in
O
mode. This would be lost upon a re-exec. And for setuids, invokingld.so
directly breaks setuid executable (though I'm not sure if this use case is something to support). And besides, saving an exec sounds cool since exec is very expensive process.Anyways, here's a demo of the POC:
The
absolute
only have minimal rpath just to show what the output looks like for a normal executable:It cannot be relocated:
And this is the relocatable:
... which is relocatable like other zapps:
Wdyt?