search

nedbat / coveragepy

The code coverage tool for Python
https://coverage.readthedocs.io
Apache License 2.0
3k stars 428 forks source link

build_ext fails when hardening options enabled #498

Closed nedbat closed 7 years ago

nedbat commented 8 years ago

Originally reported by Ben Finney (Bitbucket: bignose, GitHub: bignose)

With various “hardening” options enabled in the C compiler, the build_ext command fails with many “undefined reference” errors.

nedbat commented 7 years ago

Seems like the library is already hardened? If you can provide more information about steps I can take in building coverage.py, please feel free to re-open.

nedbat commented 7 years ago

Original comment by Thomas Wouters (Bitbucket: Yhg1s, GitHub: Yhg1s)

This problem doesn't appear to have anything to do with coverage, as such, as any extension module would have the same problem: you're (indirectly) passing -pie to the link of a shared library. -pie only makes sense on the final link of an executable, which is why it insists there is a 'main' to reference, as well as that all symbols are resolved.

Extension modules are shared libraries. They intentionally leave a lot of symbols unresolved -- all symbols that would be satisfied by the python executable they are loaded into. It's not a good idea to dynamically link extension modules to (e.g.) libpython3.5 to resolve those symbols, as that library only exists when Python is built with --enable-shared (and it might not be). It's also useless: either the libpythonX.Y shared library will already have been loaded by the python process, or it shouldn't be loaded.

The equivalent of -pie for shared libraries is -fPIC, which is already the default (because of how they work, shared libraries have to be linked position-independently, which means all names go through a runtime lookup table, just like PIE). There's also -fPIE, but I believe code generated that way can't be linked into a shared library.

nedbat commented 7 years ago

Original comment by Loic Dachary (Bitbucket: dachary, GitHub: dachary)

I only now learn about hardening, from wikipedia Position independent code and the Debian page. Although adding -lpython3.5m to the compile line at

#!bash

$ x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-Bsymbolic-functions -Wl,-z,relro -pie -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.5/coverage/ctracer/datastack.o build/temp.linux-x86_64-3.5/coverage/ctracer/filedisp.o build/temp.linux-x86_64-3.5/coverage/ctracer/module.o build/temp.linux-x86_64-3.5/coverage/ctracer/tracer.o  -lpython3.5m -o build/lib.linux-x86_64-3.5/coverage/tracer.cpython-35m-x86_64-linux-gnu.so
/usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/Scrt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: error: ld returned 1 exit status

resolves most linkage issues, it fails because main is missing, which makes sense because this is a library and not a standalone executable. I'm not sure I understand why a shared library would need pie in addition to PIC ?

Unless I missed something the shared library seems to be properly hardened by default:

#!bash

$ hardening-check build/lib.linux-x86_64-3.5/coverage/tracer.cpython-35m-x86_64-linux-gnu.so
build/lib.linux-x86_64-3.5/coverage/tracer.cpython-35m-x86_64-linux-gnu.so:
 Position Independent Executable: no, regular shared library (ignored)
 Stack protected: yes
 Fortify Source functions: unknown, no protectable libc functions used
 Read-only relocations: yes
 Immediate binding: no, not found!
nedbat commented 8 years ago

Original comment by Ben Finney (Bitbucket: bignose, GitHub: bignose)

I will attempt to reproduce this with fewer options, to narrow down the causes.

The behaviour is reproducible by requesting a position independent executable:

#!shell
$ gcc --version
gcc (Debian 5.4.0-6) 5.4.0 [20160609 (bb)](https://bitbucket.org/ned/coveragepy/commits/20160609)
[…]

$ export CFLAGS="-pie"
$ python3 ./setup.py build_ext
running build_ext
building 'coverage.tracer' extension
[…]
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,relro -pie -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.5/coverage/ctracer/datastack.o build/temp.linux-x86_64-3.5/coverage/ctracer/filedisp.o build/temp.linux-x86_64-3.5/coverage/ctracer/module.o build/temp.linux-x86_64-3.5/coverage/ctracer/tracer.o -o build/lib.linux-x86_64-3.5/coverage/tracer.cpython-35m-x86_64-linux-gnu.so
/usr/lib/gcc/x86_64-linux-gnu/5/../../../x86_64-linux-gnu/Scrt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
build/temp.linux-x86_64-3.5/coverage/ctracer/datastack.o: In function `DataStack_grow':
/home/bignose/Projects/python/coverage.py/coveragepy-4.2/coverage/ctracer/datastack.c:31: undefined reference to `PyMem_Realloc'
/home/bignose/Projects/python/coverage.py/coveragepy-4.2/coverage/ctracer/datastack.c:34: undefined reference to `PyErr_NoMemory'
build/temp.linux-x86_64-3.5/coverage/ctracer/datastack.o: In function `DataStack_dealloc':
/home/bignose/Projects/python/coverage.py/coveragepy-4.2/coverage/ctracer/datastack.c:21: undefined reference to `PyMem_Free'
build/temp.linux-x86_64-3.5/coverage/ctracer/module.o: In function `PyInit_tracer':
/home/bignose/Projects/python/coverage.py/coveragepy-4.2/coverage/ctracer/module.c:31: undefined reference to `PyModule_Create2'
/home/bignose/Projects/python/coverage.py/coveragepy-4.2/coverage/ctracer/module.c:41: undefined reference to `PyType_GenericNew'
[…]
collect2: error: ld returned 1 exit status
nedbat commented 8 years ago

Original comment by Ben Finney (Bitbucket: bignose, GitHub: bignose)

I will attempt to reproduce this with fewer options, to narrow down the causes.

Looking at many of the errors, though, I think some errors (e.g. referencing names not explicitly defined) might be from a more-strict set of checks that are active by default in recent GCC versions.

nedbat commented 8 years ago

Original comment by Ben Finney (Bitbucket: bignose, GitHub: bignose)

I don't understand what hardening is

Ah, my apologies for assuming the context. This is referring to the “hardening” of an executable against common security vulnerabilities, by enabling specific improvements – the “hardening options” – when invoking the C-language compiler.

A lot of detail can be had in this 2005 article introducing C compiler improvements. In brief, it encompasses features like stack-smashing protection, position-independent executables, buffer overflow detection, and other recent compiler features that protect against many vulnerabilities to which C code is especially prone.

why it might be enabled, or what I am supposed to do about it.

The options are enabled by the party compiling the C module, by setting environment variables that are used by the C compiler.

The resulting C compiler command line options are displayed in the attachment; you can use those options yourself to reproduce the behaviour with that compiler.

nedbat commented 8 years ago

@bignose all of the undefined references seem to be C API symbols. I don't understand what hardening is, why it might be enabled, or what I am supposed to do about it. Do you have any insight and/or guidance?