Unexpected failure when wrapping executable due to missing line in ldd output

ChrisTimperley commented 5 years ago

This project looks great. Thanks for your work!

I'm trying to wrap a dynamically linked binary generated by PyInstaller so that it can be run on systems with a different libc and ld.so. Unfortunately, I encounter the following error when I try to wrap the binary:

$ staticx bin/startcli3 portable
staticx: Unexpected line in ldd output:     libgfortran-ed201abd.so.3.0.0 => not found

$ ldd bin/startcli3
    linux-vdso.so.1 =>  (0x00007fffcdbb3000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fe074e09000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fe074bec000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe07480c000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fe07500d000)

For reference, I'm running staticx inside a virtual environment (created by pipenv) on the following platform:

$ uname -a
Linux chris-Blade 4.13.0-46-generic #51-Ubuntu SMP Tue Jun 12 12:36:29 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 17.10
Release:    17.10
Codename:   artful

ChrisTimperley commented 5 years ago

I was able to hack my way around this particular problem via:

diff --git a/staticx/elf.py b/staticx/elf.py
index 7e6f92f..aef6573 100644
--- a/staticx/elf.py
+++ b/staticx/elf.py
@@ -84,6 +84,7 @@ def get_shobj_deps(path):
     for line in output.splitlines():
         m = pat.match(line)
         if not m:
+            continue
             raise ToolError('ldd', "Unexpected line in ldd output: " + line)
         libname  = m.group(1)
         libpath  = m.group(2)

JonathonReinhart commented 5 years ago

Did your program actually run correctly with that hack in place? It seems like it would be skipping a needed library.

I'm confused by this output:

$ staticx bin/startcli3 portable
staticx: Unexpected line in ldd output:     libgfortran-ed201abd.so.3.0.0 => not found

$ ldd bin/startcli3
    linux-vdso.so.1 =>  (0x00007fffcdbb3000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fe074e09000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fe074bec000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fe07480c000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fe07500d000)

Can you explain why staticx saw ldd print libgfortran-ed201abd.so.3.0.0 => not found, but when you run ldd that library is not present?

ChrisTimperley commented 5 years ago

Surprisingly, it appears to be running correctly! not found would suggest that the library isn't to be found on the host machine, either, so presumably the wrapped binary would inherit the limitations of the host. Still, I'm baffled at why that library isn't listed by ldd. Are you calling ldd directly on the executable, or are you using another library to recursively call ldd?

JonathonReinhart commented 5 years ago

I'm just running ldd directly. If you look here, you'll see I'm referencing

tool_ldd        = ExternTool('ldd', 'binutils')

where ExternTool is just a dumb wrapper around subprocess.Popen.

I'm not doing any sort of path manipulation; just running ldd. Is there any reason staticx would see a different ldd in its PATH than you would at your shell?

rvigeant8 commented 5 years ago

Hi, I've got the same problem however I was also able to reproduce it by running ldd directly:

1- Here is what happen when running staticx: It terminates with staticx: Unexpected line in ldd output: libffi-ae16d830.so.6.0.4 => not found

(sIPve) auto-secops@secops-scripts:~/sIP$ staticx --loglevel DEBUG   ./dist/sIP ./output
DEBUG:root:Running ['patchelf', '--set-interpreter', 'iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii', '--set-rpath', 'rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr', '--force-rpath', '/tmp/staticx-prog-nh6onpqo']
INFO:root:Program interpreter: /lib64/ld-linux-x86-64.so.2
INFO:root:Using XZ BCJ filter FILTER_X86
INFO:root:Adding /tmp/staticx-prog-nh6onpqo as .staticx.prog
DEBUG:root:Running ['ldd', '/tmp/staticx-prog-nh6onpqo']
INFO:root:    Adding Symlink libdl.so.2 => libdl-2.23.so
INFO:root:    Adding /lib/x86_64-linux-gnu/libdl-2.23.so as libdl-2.23.so
INFO:root:    Adding Symlink libz.so.1 => libz.so.1.2.8
INFO:root:    Adding /lib/x86_64-linux-gnu/libz.so.1.2.8 as libz.so.1.2.8
INFO:root:    Adding Symlink libc.so.6 => libc-2.23.so
INFO:root:    Adding /lib/x86_64-linux-gnu/libc-2.23.so as libc-2.23.so
INFO:root:    Adding Symlink ld-linux-x86-64.so.2 => ld-2.23.so
INFO:root:    Adding /lib/x86_64-linux-gnu/ld-2.23.so as ld-2.23.so
INFO:root:Opened PyInstaller archive!
DEBUG:root:Extracting to /tmp/staticx-pyi-26d27yzf/_bz2.cpython-35m-x86_64-linux-gnu.so
DEBUG:root:Running ['ldd', '/tmp/staticx-pyi-26d27yzf/_bz2.cpython-35m-x86_64-linux-gnu.so']
INFO:root:    Adding Symlink libpthread.so.0 => libpthread-2.23.so
INFO:root:    Adding /lib/x86_64-linux-gnu/libpthread-2.23.so as libpthread-2.23.so
DEBUG:root:libc.so.6 already in staticx archive
DEBUG:root:libbz2.so.1.0 already in pyinstaller archive
DEBUG:root:ld-linux-x86-64.so.2 already in staticx archive
DEBUG:root:Extracting to /tmp/staticx-pyi-26d27yzf/_cffi_backend.cpython-35m-x86_64-linux-gnu.so
DEBUG:root:Running ['ldd', '/tmp/staticx-pyi-26d27yzf/_cffi_backend.cpython-35m-x86_64-linux-gnu.so']
staticx: Unexpected line in ldd output:         libffi-ae16d830.so.6.0.4 => not found
(sIPve) auto-setops@setops-scripts:~/sIP$

2- It works when running ldd against the library in its original location:

(sIPve) auto-setops@setops-scripts:~/sIP$ ldd /home/auto-setops/sIP/sIPve/lib/python3.5/site-packages/_cffi_backend.cpython-35m-x86_64-linux-gnu.so
        linux-vdso.so.1 =>  (0x00007ffe86564000)
        libffi-ae16d830.so.6.0.4 => /home/auto-setops/sIP/sIPve/lib/python3.5/site-packages/.libs_cffi_backend/libffi-ae16d830.so.6.0.4 (0x00007fb3792fb000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fb3790de000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb378d14000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fb37973a000)
(sIPve) auto-setops@setops-scripts:~/sIP$

3- It fails when running ldd against the same library after it was copied to the tmp directory:

(sIPve) auto-setops@setops-scripts:~/sIP$ ldd  /tmp/testldd/_cffi_backend.cpython-35m-x86_64-linux-gnu.so
        linux-vdso.so.1 =>  (0x00007fffaa186000)
        libffi-ae16d830.so.6.0.4 => not found
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fee8293e000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fee82574000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fee82d91000)
(sIPve) auto-setops@setops-scripts:~/sIP$

rvigeant8 commented 5 years ago

I will continue investigate with ldd

JonathonReinhart commented 5 years ago

Okay, the difference between (2) and (3) seems to be the culprit: When the library is moved, one if its dependencies cannot be found.

I'm guessing _cffi_backend.cpython-35m-x86_64-linux-gnu.so sets its RPATH to $ORIGIN, which means "look in the same directory as this .so for other .so files." So when staticx copies it to the temp directory, the dependencies in the same directory can no longer be found.

Can you confirm this by posting the output of readelf -d /original/path/to/_cffi_backend.cpython-35m-x86_64-linux-gnu.so? Specifically, I'm looking for (RPATH).

This should be pretty easy to fix in staticx. I just need to refactor things so we run ldd on the original file path, and not the temp path after it's been copied.

rvigeant8 commented 5 years ago

Sorry I was out of the office.

The result of "readelf -d" is:

readelf -d sIPve/lib/python3.5/site-packages/_cffi_backend.cpython-35m-x86_64-linux-gnu.so

Dynamic section at offset 0xc9000 contains 28 entries:
  Tag        Type                         Name/Value
 0x000000000000000f (RPATH)              Library rpath: [$ORIGIN/.libs_cffi_backend]
 0x0000000000000001 (NEEDED)             Shared library: [libffi-ae16d830.so.6.0.4]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [ld-linux-x86-64.so.2]
 0x000000000000000c (INIT)               0x6ac0
 0x000000000000000d (FINI)               0x214b8
 0x0000000000000019 (INIT_ARRAY)         0x229270
 0x000000000000001b (INIT_ARRAYSZ)       8 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x229278
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x232ee0
 0x0000000000000005 (STRTAB)             0x233200
 0x0000000000000006 (SYMTAB)             0x2340f8
 0x000000000000000a (STRSZ)              3831 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000003 (PLTGOT)             0x229d48
 0x0000000000000002 (PLTRELSZ)           4104 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x5ab8
 0x0000000000000007 (RELA)               0x2890
 0x0000000000000008 (RELASZ)             12840 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffe (VERNEED)            0x2820
 0x000000006fffffff (VERNEEDNUM)         3
 0x000000006ffffff0 (VERSYM)             0x2654
 0x000000006ffffff9 (RELACOUNT)          474
 0x0000000000000000 (NULL)               0x0

haizaar commented 5 years ago

Ran into this issue as well while trying to run staticx on executable produced by PyInstaller. The above suggestion did help to make it work.

JonathonReinhart commented 5 years ago

@haizaar Thanks for the feedback. It's great to see this project is getting some use!

The continue suggestion is good as a quick test, but it should be considered a temporary hack and not a permanent solution. Adding continue there will allow staticx to build an incomplete output executable.

I'll get back to #75 soon to finish the real fix.

danielguardicore commented 5 years ago

@JonathonReinhart I'm also trying to use the project, same use case of Pyinstaller. @haizaar did the binary run after you inserted the workaround?

haizaar commented 5 years ago

@danielguardicore yes, it did work.

But it didn't work later on for these guys: https://github.com/crossbario/crossbar/issues/1428#issuecomment-453743862

JonathonReinhart commented 5 years ago

I've finally found some time to come back to this issue, and I've narrowed down a simple test case (see the add-pyinstall-cff-test branch).

I'm specifically working in a virtualenv (named venv) with the following relevant versions:

$ python --version
Python 3.6.7
$ pip freeze
cffi==1.12.2
...
PyInstaller==3.4

My fixes thus far in #75 are correct, but insufficient for PyInstaller apps. As I noted in https://github.com/JonathonReinhart/staticx/pull/75#issuecomment-439766244, the problem seems to be that staticx runs ldd on the shared object it extracted from the PyInstaller archive to a temporary directory:

DEBUG:root:Extracting to /tmp/staticx-pyi-elnfmwc6/_cffi_backend.cpython-36m-x86_64-linux-gnu.so
DEBUG:root:Running ['ldd', '/tmp/staticx-pyi-elnfmwc6/_cffi_backend.cpython-36m-x86_64-linux-gnu.so']
staticx: Unexpected line in ldd output:     libffi-ae16d830.so.6.0.4 => not found

Most of the time, this would probably work okay, but _cffi_backend.cpython-36m-x86_64-linux-gnu.so is using RPATH:

$ readelf -d venv/lib/python3.6/site-packages/_cffi_backend.cpython-36m-x86_64-linux-gnu.so 

Dynamic section at offset 0xcc000 contains 28 entries:
  Tag        Type                         Name/Value
 0x000000000000000f (RPATH)              Library rpath: [$ORIGIN/.libs_cffi_backend]
 0x0000000000000001 (NEEDED)             Shared library: [libffi-ae16d830.so.6.0.4]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [ld-linux-x86-64.so.2]

Here's directory structure where it is installed:

venv/lib/python3.6/site-packages
├── cffi
│   ├── _cffi_errors.h
│   ├── _cffi_include.h
│   ├── cffi_opcode.py
│   └── __pycache__
│       └── cffi_opcode.cpython-36.pyc
├── cffi-1.12.2.dist-info
├── _cffi_backend.cpython-36m-x86_64-linux-gnu.so
└── .libs_cffi_backend
    └── libffi-ae16d830.so.6.0.4

So at first glance, it looks like I should simply ensure to extract libffi-ae16d830.so.6.0.4 to a .libs_cffi_backend sub-directory next to _cffi_backend.cpython-36m-x86_64-linux-gnu.so before calling ldd on the latter.

The problem is it doesn't appear that PyInstaller is able to give me that information. Here are the relevant lines from pyi-archive_viewer:

 pos, length, uncompressed, iscompressed, type, name
[(0, 270, 348, 1, 'm', 'struct'),
 (270, 1123, 1837, 1, 'm', 'pyimod01_os_path'),
 (1393, 4379, 9397, 1, 'm', 'pyimod02_archive'),
 (5772, 7402, 18702, 1, 'm', 'pyimod03_importers'),
 (13174, 1847, 4157, 1, 's', 'pyiboot01_bootstrap'),
 (15021, 1077, 1774, 1, 's', 'pyi_rth_multiprocessing'),
 (16098, 210, 254, 1, 's', 'pyi_rth_pkgres'),
 (16308, 331, 462, 1, 's', 'app'),
...
 (55536, 281114, 845576, 1, 'b', '_cffi_backend.cpython-36m-x86_64-linux-gnu.so'),
...
 (2792700, 52534, 149064, 1, 'b', 'libffi-ae16d830.so.6.0.4'),
 (2845234, 17803, 38544, 1, 'b', 'libffi.so.6'),
...
 (5628336, 2184807, 2184807, 0, 'z', 'PYZ-00.pyz')]

Note that there is no .libs_cffi_backend subdirectory visible for libffi-ae16d830.so.6.0.4. In fact, if we look in the _MEIxxxxxx temporary directory when the application is (successfully) running, we can see that the PyInstaller bootloader hasn't even extracted it "correctly" (the structure is "flat", just as I have extracted it):

$ cd /tmp/_MEI3u5Uge
$ ls -1 *ffi*
_cffi_backend.cpython-36m-x86_64-linux-gnu.so
libffi-ae16d830.so.6.0.4
libffi.so.6
$ ldd _cffi_backend.cpython-36m-x86_64-linux-gnu.so 
    linux-vdso.so.1 (0x00007ffdbedf5000)
    libffi-ae16d830.so.6.0.4 => not found
    libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x00007fa02f284000)
    libc.so.6 => /usr/lib64/libc.so.6 (0x00007fa02eec5000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fa02f6da000)

At this point, I'm not even sure how ld.so even loads this PyInstaller application correctly when it runs! :flushed:

So it seems that PyInstaller isn't giving me enough info to know the correct path structure. I need to keep digging.

JonathonReinhart commented 5 years ago

Whether it's right or wrong, PyInstaller will flatten the shared object directory structure when building its archive. The reason the application runs correctly is because (on Linux), the PyInstaller bootloader points LD_LIBRARY_PATH at the _MEIxxxxxx path:

https://github.com/pyinstaller/pyinstaller/blob/v3.4/bootloader/src/pyi_utils.c#L808-L811

Indeed, setting LD_LIBRARY_PATH makes my previous test work:

$ cd /tmp/_MEIMszltz
$ LD_LIBRARY_PATH="$(pwd):$LD_LIBRARY_PATH" ldd _cffi_backend.cpython-36m-x86_64-linux-gnu.so
    linux-vdso.so.1 (0x00007fff16b22000)
    libffi-ae16d830.so.6.0.4 => /tmp/_MEIMszltz/libffi-ae16d830.so.6.0.4 (0x00007fefebe70000)
    libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x00007fefebc51000)
    libc.so.6 => /usr/lib64/libc.so.6 (0x00007fefeb892000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fefec2b0000)

I think I can apply this logic in staticx to resolve our issues.

JonathonReinhart commented 5 years ago

Hey everyone,

I went ahead and merged #75 without waiting on any feedback because:

I definitely identified several issues and fixed them
I'm fairly confident my test cases cover the problematic scenarios you all have presented
It's really late, and I don't want to drag this issue out any longer :smile:

Hopefully this resolves at least some of the problems you were having with staticx + PyInstaller, let me know if it works for you! I appreciate your patience and your interest in my little project. If you have any further problems, please open an issue!

If you'd like to try it out, you can install this pre-release from TestPyPI: staticx 0.6.0.182

~~I'll be releasing a new version to PyPI soon.~~ I've released staticx v0.7.0 and it will be available on PyPI momentarily.

cc @cakebake @oz123

danielguardicore commented 5 years ago

So nearly there, but still no cigar. Thank you for all the work.

In this case, it's failing when PyInstaller bundled an additional binary file. My specific case has 2 binary files added, this is the only one causing a problem.

DEBUG:root:Running ['ldd', u'/tmp/staticx-pyi-RjKceH/traceroute64'] staticx: ldd returned 1

Running ldd manually shows that ldd thinks this isn't a dynamic executable.

Would you like me to upload the debug log?

JonathonReinhart commented 5 years ago

@danielguardicore Yes, please. Would you mind opening a new issue? Please include:

The output of staticx --loglevel DEBUG ...
The output of pyi-archive_viewer yourapp
Any details about traceroute64, and why/how it's included in your PyInstaller app

Thanks!

danielguardicore commented 5 years ago

Opened #78 thank you

JonathonReinhart / staticx

Unexpected failure when wrapping executable due to missing line in ldd output #61