patchelf 0.16.0 or later causes staticx failures or SIGSEGV

haboustak commented 4 months ago

I've recently ran into problems running a python script processed with pyinstaller and staticx when run on Debian 10 (Buster), Linux kernel 4.19. The goal of this meta-issue is to collect several suspected patchelf problems idenfied in other staticx issues and resolve them here.

TLDR:

patchelf 0.15.0 or earlier: Binaries work on Debian 10 (4.19.y) and Debian 12 (6.1.y)
patchelf 0.16.0 - 0.17.0: Binaries SEGFAULT on some Linux kernel versions (Debian 10, 4.19.y)
patchelf 0.17.1 or later: staticx fails to run with patchelf assertion

In the 0.16.0 => 0.17.0 range, patchelf creates sections that the kernel ELF loader is not happy with. Through a mixture of strace and dmesg I was able to build this execution log of the failing binary. I think the problem occurs when the staticx binary tries to exec the wrapped executable (in this case, my pyinstaller'd script).

[pid  2326] execve("/tmp/staticx-MMpalG/art", ["/tmp/staticx-MMpalG/art"], 0x7f0ef252c7d0 /* 4 vars */ <unfinished ...>
[pid    30] rt_sigaction(SIGTERM, {sa_handler=0x401c10, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x408dcf}, NULL, 8) = 0
[pid    30] wait4(2326,  <unfinished ...>
[pid  2326] <... execve resumed>)       = -1 EEXIST (File exists)
[pid  2326] --- SIGSEGV {si_signo=SIGSEGV, si_code=SI_KERNEL, si_addr=NULL} ---
[2138743.980717] 21481 (art): Uhuuh, elf segment at 00000000003ff000 requested but the memory is mapped already
[pid  2326] +++ killed by SIGSEGV +++

This trace led me to a patchelf commit that was introduced in 0.17.1

Fix bug in file shifting that could cause conflicting PT_LOAD segments

https://github.com/NixOS/patchelf/commit/8d2cb4f9ab8d564904c292099a022ffb3cccd52d

The logic that triggers the kernel crash was added to patchelf in 0.16.0. It was fixed in patchelf 0.17.1. The kernel crash does not occur when patchelf 0.15.0 is used.

Some aspect of the changes in commit 8d2cb4f prevent combining the --set-interpreter, --set-rpath, and --no-default-lib options in the same invocation of patchelf. I am not sure yet what the nature of the change is, but the adjustments to the shiftFile arguments has thrown off the validation / segment matching when it comes to these cascading section/segment resizes.

$ ./patchelf-8d2cb4f --set-interpreter "iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii" \
    --set-rpath "rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr" \
    --force-rpath \
    --no-default-lib \
    pyart \
    ; echo $?
patchelf-8d2cb4f: patchelf.cc:508: void ElfFile<Elf_Ehdr, Elf_Phdr, Elf_Shdr, Elf_Addr, Elf_Off, Elf_Dyn, Elf_Sym, Elf_Verneed, Elf_Versym>::shiftFile(unsigned int, size_t, size_t) [with Elf_Ehdr = Elf64_Ehdr; Elf_Phdr = Elf64_Phdr; Elf_Shdr = Elf64_Shdr; Elf_Addr = long unsigned int; Elf_Off = long unsigned int; Elf_Dyn = Elf64_Dyn; Elf_Sym = Elf64_Sym; Elf_Verneed = Elf64_Verneed; Elf_Versym = short unsigned int; size_t = long unsigned int]: Assertion `splitIndex != -1' failed.
Aborted
134

$ ./patchelf-f4f1848 --set-interpreter "iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii" \
    --set-rpath "rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr" \
    --force-rpath \
    --no-default-lib \
    pyart \
    ; echo $?
0

It's some kind of alignment issue, because if I use smaller values for either interpreter or rpath, patchelf no longer asserts.

$ ./patchelf-8d2cb4f --set-interpreter "i" --set-rpath "r" --force-rpath --no-default-lib pyart ; echo $?
0

If I run --no-default-lib first, patchelf succeeds. If I set interpreter and rpath first, I cannot set --no-default-lib in a later invocation.

$ ./patchelf-8d2cb4f --no-default-lib pyart ; echo $?
0

$ ./patchelf-8d2cb4f --set-interpreter "iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii" \
    --set-rpath "rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr" \
    --force-rpath \
    pyart \
    ; echo $?
0

$ readelf -d pyart 

Dynamic section at offset 0xff8 contains 29 entries:
  Tag        Type                         Name/Value
 0x000000000000000f (RPATH)              Library rpath: [rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr]
 0x000000006ffffffb (FLAGS_1)            Flags: NODEFLIB

haboustak commented 4 months ago

I am working on creating an upstream patchelf issue. I think this may be fixed in 0.18.0, but that release was yanked from PyPI. Maybe because of failing tests?

$ ./patchelf-99c2423 \
    --set-interpreter "iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii" \
    --set-rpath "rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr" \
    --force-rpath \
    --no-default-lib 
    pyart \
    ; echo $?
0

I think there are 55 non-merge commits to bisect.

$ git log --format=oneline --no-merges 0.17.2..0.18.0 | wc -l
55

haboustak commented 4 months ago

git-bisect identified the following commit as required for staticx compatibility

$ git bisect new
65cdee904431d16668f95d816a495bc35a05a192 is the first new commit
commit 65cdee904431d16668f95d816a495bc35a05a192
Author: Breno Rodrigues Guimaraes <brenorg@gmail.com>
Date:   Mon Mar 20 07:48:00 2023 -0300

    Resize segment mapping rewritten sections if needed

https://github.com/NixOS/patchelf/pull/485

Unfortunately, this is in patchelf 0.18.0, which has been yanked from PyPI and has been rolled back by several dependent projects due to this open bug: https://github.com/NixOS/patchelf/issues/492

JonathonReinhart / staticx

patchelf 0.16.0 or later causes staticx failures or SIGSEGV #285