UPX "sometimes" failed to pack amd64 PIE executables

upx / upx

UPX - the Ultimate Packer for eXecutables

https://upx.github.io

Other

14.2k stars 1.34k forks source link

UPX "sometimes" failed to pack amd64 PIE executables #166

Closed ThinerDAS closed 6 years ago

ThinerDAS commented 6 years ago

What's the problem (or question)?

I cannot use upx to compress executables that is PIE, for example library and ld. The phenomenon is inconsistent between versions and between binaries, sometimes it fails to compress (complain that DT_INIT is missing or, just refuse to compress), and sometimes it fails to execute+uncompress.

I have used docker "frolvlad/alpine-gxx" to generate some pie static executables ("gcc -static -Os -s"/"g++ -static -Os -s").

test1.zip hello and acm2 uses "frolvlad/alpine-gxx" while acm1 uses the compiler in Ubuntu 16.04 ld and libc are from Ubuntu 16.04.

Of all PIE executables, only that compiled with "g++ -fPIE -pie" on ubuntu 16.04 passed the test fully. (that is, acm1)

What should have happened?

The upx should compress the executable successfully, no matter whether the executable is PIE, static, with or without DT_INIT (in some cases compiled binary just does not have the symbol and adding the line will make the compiler complain, a signal that the compiler does provide _init but does not use DT_INIT).

Do you have an idea for a solution?

I am not sure but my observation is that some packed executables that crash, the header is the same as original and is mismatched since it is packed. I don't know more on the issue.

How can we reproduce the issue?

Use upx to compress one mentioned executable
Execute it
Expect either upx refuses to compress the executable, or the executable exits with SIGSEGV.

Please tell us details about your environment.

UPX version used (upx --version): 3.92, 3.94, development release
Host Operating System and version: Ubuntu 16.04, kernel 4.9.65-040965-generic, yet some compilers are in docker so my preference on host OS is unimportant, only ELF is important.
Host CPU architecture: amd64
Target Operating System and version: Same as above, or, what is the question...
Target CPU architecture: amd64

jreiser commented 6 years ago

@ThinerDAS The assumptions that are made by the runtime environment that invokes a compressed ET_DYN file differ (execve() vs dlopen()), and there is no unification of the assumptions that is reasonable, so at compress time then upx must know which is which. If the PT_DYNAMIC contains DT_FLAGS_1 with DF_1_PIE, then upx believes it is a main program that will be invoked directly by execve(). If no DF_1_PIE, then upx believes it is a shared library that will be invoked by dlopen(). [Note that being designated as DT_NEEDED means an implicit dlopen().] The software tool chain (especially the static binder [/bin/ld]) which builds the ET_DYN must set the flag correctly.

Edit: and if a shared library, then it must have a DT_INIT because that's the hook for de-compressing. The hook for execve is ElfXX_Ehdr.e_entry.

The "rtld" runtime linker ld-linux is even more special: there are 3 possible invocation methods: as a shared library [unusual but possible], as a main program [via /usr/bin/ldd, or directly via top-level execve() in order to use --library-path for the purpose of setting LD_LIBRARY_PATH for the top-level process but not for its children], or [most commonly] as the PT_INTERP during execve(). Compression can accommodate only one of these methods at a time. Use the upx command-line flag --is_ptinterp if that's what you want, otherwise upx will assume a shared library because there is no DF_1_PIE. My best advice is: do not compress ld-linux.

In the test1.zip then acm1 has DF_1_PIE, and is treated as a main program, and works for me when compressed with the devel branch. acm2 also works for me when compressed using the devel branch. I get equivalent output from strace, whether compressed or uncompressed: all 4 executables are waiting for input from stdin.

ThinerDAS commented 6 years ago

$ sha256sum acm2
35529f51bc15f89bc585d55225ad20b75e66232fc9faf5682d5b2e8435e08bf9  acm2
$ ldd acm2
    statically linked
$ ./upx.out acm2 -o acm21
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2017
UPX git-d31947  Markus Oberhumer, Laszlo Molnar & John Reiser   May 12th 2017

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
    678824 ->    287916   42.41%   linux/amd64   acm21                         

Packed 1 file.

WARNING: this is an unstable beta version - use for testing only! Really.
$ ./acm21
Segmentation fault (core dumped)
$ uname
Linux
$ uname -a
Linux thiner-ThinkPad-T440p 4.9.65-040965-generic #201711240331 SMP Fri Nov 24 08:33:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
$ strace ./acm21
execve("./acm21", ["./acm21"], [/* 73 vars */]) = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x7fab5aa0a295} ---
+++ killed by SIGSEGV (core dumped) +++
Segmentation fault
$

mmm, please let me know what I did wrong ...

jreiser commented 6 years ago

On the devel branch, the binary version

UPX git-d31947  Markus Oberhumer, Laszlo Molnar & John Reiser   May 12th 2017

[Note the May 12th] has some connection to the source commit

commit d31947e1f016e87f24f88b944439bbee892f0429
Author: Markus F.X.J. Oberhumer <markus@oberhumer.com>
Date:   Fri May 12 13:01:20 2017 +0200

    Update NEWS.

which is 7 months old. [If in doubt: make clean; make]

I used

UPX git-e04bf9+ Markus Oberhumer, Laszlo Molnar & John Reiser   May 13th 2017

[Note the May 13th which is after May 12] corresponding to

commit e04bf9e4bcec1ede9e5e123d11b17ac28b706b9d
Author: John Reiser <jreiser@BitWagon.com>
Date:   Thu Dec 28 17:40:04 2017 -0800

    more checking of PT_DYNAMIC

    https://github.com/upx/upx/issues/164
            modified:   p_lx_elf.cpp
            modified:   p_lx_elf.h

which is current.

$ ../upx.out -f -o foo acm2
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2017
UPX git-e04bf9+ Markus Oberhumer, Laszlo Molnar & John Reiser   May 13th 2017

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
    678824 ->    217452   32.03%   linux/amd64   foo                           

Packed 1 file.

WARNING: this is an unstable beta version - use for testing only! Really.
$ sha256sum acm2 foo
35529f51bc15f89bc585d55225ad20b75e66232fc9faf5682d5b2e8435e08bf9  acm2
d01674df4b4cac5bd05d7c029c7734b0f7ed8b448a3ebfd5c1d5f30e2c5ee575  foo
$ strace ./foo
execve("./foo", ["./foo"], 0x7ffccea8f070 /* 63 vars */) = 0
open("/proc/self/exe", O_RDONLY)        = 3
mmap(NULL, 217168, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7efe57ba3000
mmap(0x7efe57ba3000, 216810, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0x7efe57ba3000
mprotect(0x7efe57bd7000, 4176, PROT_READ|PROT_EXEC) = 0
readlink("/proc/self/exe", "/bigdata/home2/upx/src/github-is"..., 4095) = 42
mmap(0x7efe57bd9000, 2797568, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7efe57bd9000
mmap(0x7efe57bd9000, 646368, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7efe57bd9000
mprotect(0x7efe57bd9000, 646368, PROT_READ|PROT_EXEC) = 0
mmap(0x7efe57e77000, 29832, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0x9e000) = 0x7efe57e77000
mprotect(0x7efe57e77000, 29832, PROT_READ|PROT_WRITE) = 0
mmap(0x7efe57e7f000, 16592, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7efe57e7f000
mmap(NULL, 4096, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7efe57ba2000
close(3)                                = 0
munmap(0x7efe57ba3000, 217168)          = 0
arch_prctl(ARCH_SET_FS, 0x7efe57e82da8) = 0
set_tid_address(0x7efe57e82de0)         = 16887
brk(NULL)                               = 0x7efe59880000
brk(0x7efe59892000)                     = 0x7efe59892000
readv(0, ^C[{iov_base=0x7fffc6b068af, iov_len=0}, {iov_base=0x7efe57e81ce8, iov_len=1024}], 2) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
strace: Process 16887 detached

ThinerDAS commented 6 years ago

I am not sure if I missed something. But when I tried git checkout -t origin/devel and make clean;make all (make seems not to work) it is a bit different from what you have shown:

$ ./upx.out --force acm2 -o foo
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2017
UPX git-e04bf9  Markus Oberhumer, Laszlo Molnar & John Reiser   May 13th 2017

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
upx.out: acm2: CantPackException: need DT_INIT; try "void _init(void){}"

Packed 0 files.

WARNING: this is an unstable beta version - use for testing only! Really.
$ ./upx.out -f acm2 -o foo
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2017
UPX git-e04bf9  Markus Oberhumer, Laszlo Molnar & John Reiser   May 13th 2017

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
upx.out: acm2: CantPackException: need DT_INIT; try "void _init(void){}"

Packed 0 files.

WARNING: this is an unstable beta version - use for testing only! Really.

Which is same as what I have observed in the downloaded binary of your binary build (namely DT_INIT error). The difference between what you have posted is the line:

UPX git-e04bf9+ Markus Oberhumer, Laszlo Molnar & John Reiser   May 13th 2017

You see, on my machine it was

UPX git-e04bf9  Markus Oberhumer, Laszlo Molnar & John Reiser   May 13th 2017

a bit different, right? By the way using the devel version on the busybox executable (which is non PIE and static) crashed the resulting executable on return:

$ ./busybox whoami
thiner
$ upx busybox -o whoami
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2017
UPX 3.94        Markus Oberhumer, Laszlo Molnar & John Reiser   May 12th 2017

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
   1964536 ->    860700   43.81%   linux/amd64   whoami                        

$ ./whoami
thiner
$ rm whoami
$ ./upx.out busybox -o whoami
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2017
UPX git-e04bf9  Markus Oberhumer, Laszlo Molnar & John Reiser   May 13th 2017

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
   1964536 ->    923900   47.03%   linux/amd64   whoami                        

Packed 1 file.

WARNING: this is an unstable beta version - use for testing only! Really.
$ ./whoami
thiner
Segmentation fault (core dumped)
$

So you can try inputting "3 1 2 3" to acm2 and see what happens?

jreiser commented 6 years ago

The SIGSEGV for static uClibc should be fixed by the commit above.

upx.out: acm2: CantPackException: need DT_INIT; try "void _init(void){}"

Please use git pull on the devel branch before running make clean; make all.

$ ../upx.out -f -o foo acm2
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2017
UPX git-e04bf9+ Markus Oberhumer, Laszlo Molnar & John Reiser   May 13th 2017

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
    678824 ->    217452   32.03%   linux/amd64   foo                           

Packed 1 file.

WARNING: this is an unstable beta version - use for testing only! Really.
$ sha256sum acm2 foo
35529f51bc15f89bc585d55225ad20b75e66232fc9faf5682d5b2e8435e08bf9  acm2
d797830a28086b7d664d6cce6927b6953dfa4d634459c5ffd843404ea1d6054c  foo
$ file foo
foo: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), statically linked, stripped
$ ./foo
3 1 2 3
3 2 1 
$

ThinerDAS commented 6 years ago

Both my build and your build complain about DT_INIT.

$ ./upx-git-4a35bf32eac0.out -f -o foo acm2
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2017
UPX git-4a35bf  Markus Oberhumer, Laszlo Molnar & John Reiser   May 13th 2017

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
upx-git-4a35bf32eac0.out: acm2: CantPackException: need DT_INIT; try "void _init(void){}"

Packed 0 files.

WARNING: this is an unstable beta version - use for testing only! Really.
$ ./upx.out -f -o foo acm2
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2017
UPX git-4a35bf  Markus Oberhumer, Laszlo Molnar & John Reiser   May 13th 2017

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
upx.out: acm2: CantPackException: need DT_INIT; try "void _init(void){}"

Packed 0 files.

$ sha256sum upx-git-4a35bf32eac0.out 
9cfd31ef56fae3e9a31bfa0b4711c9154e3a866a17a19c97a15a743e74e7ae97  upx-git-4a35bf32eac0.out
$ sha256sum acm2
35529f51bc15f89bc585d55225ad20b75e66232fc9faf5682d5b2e8435e08bf9  acm2

jreiser commented 6 years ago

valgrind found an uninit member variable. Fixed on devel branch.

ThinerDAS commented 6 years ago

Awesome! With the latest build I tried both amd64 and i386 PIE static executables (all compiled with docker alpine gcc/g++ with g++ -Os -static acm.cpp -o acm), and using upx-git-507e19945eae.out both works as expected, and there seems to be nothing wrong from here. For future reference:

$ file acm32 acm64
acm32: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, stripped
acm64: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, stripped
$ ./upx-git-507e19945eae.out acm32 -o acm32x -9 --ultra-brute
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2017
UPX git-507e19  Markus Oberhumer, Laszlo Molnar & John Reiser   May 13th 2017

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
    735104 ->    188132   25.59%   linux/i386    acm32x                        

Packed 1 file.

WARNING: this is an unstable beta version - use for testing only! Really.
$ ./upx-git-507e19945eae.out acm64 -o acm64x -9 --ultra-brute
                       Ultimate Packer for eXecutables
                          Copyright (C) 1996 - 2017
UPX git-507e19  Markus Oberhumer, Laszlo Molnar & John Reiser   May 13th 2017

        File size         Ratio      Format      Name
   --------------------   ------   -----------   -----------
    678824 ->    170644   25.14%   linux/amd64   acm64x                        

Packed 1 file.

WARNING: this is an unstable beta version - use for testing only! Really.
$ ./acm32x
4 1 2 3 4
4 3 2 1 
$ ./acm64x
4 2 3 4 5
5 4 3 2 
$

Thank you for your help and patience! @jreiser

jreiser commented 6 years ago

Please git pull on the devel branch, re-make upx (in particular src/stub/amd64-linux.elf-fold.h is new), and re-compress any programs. The upx runtime de-compression stub did not keep the stack 16-byte aligned, and there will be problems with code generated by gcc 7.x that uses the movaps opcode (and others) to manipulate consecutive pointers in on-stack local structs. In particular, dlopen() and others will get SIGSEGV.