Open ComputerDruid opened 5 months ago
Side note: I probably could have gotten this to work with a dylib instead of a cdylib, but I was really struggling to get it to work at all and figured cdylibs were probably more widely used. Even so I could never get -Crpath=yes
to help here and had to resort to using patchelf
to set DT_RUNPATH
to what I wanted.
Thanks for reporting this! I will be the first to admit that while I know the basics of dynamic linkage, it is a black magic that I try not to dabble with.
Just to summarize:
/deps/<dylib>
when looking for the dylib and fails because that doesn't exist.So I think that, as you mentioned, setting the LD_ORIGIN_PATH environment variable should probably work - did you try setting it in the Isolate::run
call's env vars map? Incidentally I really should change the interface so that it only takes the isolate name and you can optionally add env vars like you can bind mounts with Isolate::new
Another option would be to bindmount the dylib to /deps/<dylib>
if you know ahead of time you or some lib you're using is doing this, but at that point you might as well just fix it ahead of time with patchelf (i.e. un-dynamicize the rpath).
If LD_ORIGIN_PATH works, I think it would probably be best if I made Isolate::run
return a builder and then you or I could add a method like enable_dylib_ld_origin()
or something that's just a wrapper for setting LD_ORIGIN_PATH
. Writing a test for it is going to be a pain and might be the impetus for splitting out isolate_test.rs
into a separate crate.
"Regular" dynamic libraries aren't affected because their path is essentially hardcoded
Basically yes, it uses the system search path which is not relative to $ORIGIN
. Only "bundled" libraries would be affected by the copy to the memfd.
But yes, your summary seems accurate.
Another option would be to bindmount the dylib to
/deps/<dylib>
if you know ahead of time you or some lib you're using is doing this, but at that point you might as well just fix it ahead of time with patchelf (i.e. un-dynamicize the rpath).
If you put an absolute path in DT_RUNPATH
you wouldn't be able to move the binary+library to a different directory. In other words you can't distribute it in a tarball. And you can't rely on the bindmount to put it in some arbitrary path because ld.so
needs to find it for the binary outside the isolate, too. So you'd have to use both DT_RUNPATH
with $ORIGIN
and a bindmount. I'd hesitate to rely on it looking in /deps/
though. I think it got that because dirname "/memfd:isolate_memfd (deleted)"
gives /
but that doesn't seem like something to rely on. You can put multiple entries in DT_RUNPATH
so I guess you could put both an $ORIGIN
-relative path and an arbitrary absolute one for the bindmount.
did you try setting it in the
Isolate::run
call's env vars map?
OK, I just tried that, and it did not seem to work. Specifically I tried
let mut env = HashMap::new();
env.insert("LD_ORIGIN_PATH".to_owned(), "/home/cdruid/src/extrasafe_example/example/target/debug".to_owned());
let output = Isolate::run("isolate_print", &env).unwrap();
and got
[pid 32252] execve("/proc/self/fd/3", ["isolate_print"], ["LD_ORIGIN_PATH=/home/cdruid/src/extrasafe_example/example/target/debug"] <unfinished ...>
...
[pid 32252] <... execve resumed>) = 0
[pid 32252] brk(NULL) = 0x56526e4d0000
[pid 32252] mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fde6d20d000
[pid 32252] readlinkat(AT_FDCWD, "/proc/self/exe", "/memfd:isolate_memfd (deleted)", 4096) = 30
[pid 32252] access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
[pid 32252] openat(AT_FDCWD, "//deps/glibc-hwcaps/x86-64-v3/libdylib.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 32252] newfstatat(AT_FDCWD, "//deps/glibc-hwcaps/x86-64-v3/", 0x7ffc2fcef3c0, 0) = -1 ENOENT (No such file or directory)
[pid 32252] openat(AT_FDCWD, "//deps/glibc-hwcaps/x86-64-v2/libdylib.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 32252] newfstatat(AT_FDCWD, "//deps/glibc-hwcaps/x86-64-v2/", 0x7ffc2fcef3c0, 0) = -1 ENOENT (No such file or directory)
[pid 32252] openat(AT_FDCWD, "//deps/libdylib.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 32252] newfstatat(AT_FDCWD, "//deps/", 0x7ffc2fcef3c0, 0) = -1 ENOENT (No such file or directory)
Which is the same behavior despite LD_ORIGIN_PATH
being set in the execve
syscall.
Apparently according to https://www.technovelty.org/linux/exploring-origin.html LD_ORIGIN_PATH
is actually a fallback if readlink("/proc/self/exe")
fails, unfortunately for us.
To be clear: for me this is an academic exercise; I simply got curious while reading your blog post and wanted to share what I learned. This bug isn't blocking me or anything like that.
Apparently according to https://www.technovelty.org/linux/exploring-origin.html LD_ORIGIN_PATH is actually a fallback if readlink("/proc/self/exe") fails, unfortunately for us.
That is unfortunate. I think probably the best thing to do if someone were in this situation would just be to use patchelf to fix the library path to not be dynamic.
Anyway, thanks for reading my blog post and teaching me about $ORIGIN!
While reading the blog post at https://harrystern.net/extrasafe-user-namespaces.html I thought to myself: that doesn't sound like it would work, because it would mess up how binaries find their shared libraries when they set a
DT_RUNPATH
with$ORIGIN
in it.I thought it would be quick to check this by whipping up a quick paired dylib + binary crate, but I was very wrong about how quick it would be.
But after hours of futzing with things, I managed to get a binary which the dynamic loader loads fine outside the isolated environment, but fails inside:
It goes like:
Then we just need some final hackery to set up
DT_RUNPATH
:Which gives me:
and looking inside
strace -f
I can see the working open at the top:followed by the broken one down below:
Looking through
man ld.so
, it looks like the fix might be as simple as settingLD_ORIGIN_PATH
based ondirname $(readlink /proc/self/exe)