Open xanderdunn opened 2 years ago
It looks like all of the undefined references are system calls: dlsym
, pthread_attr_init
, pthread_getattr_np
, pthread_attr_destroy
, getrusage
, syslog
, madvise
, setrlimit
, isatty
, sigaltstack
, pipe
, execv
, dlerror
, getauxval
.
Some of these I think I may be able to manually implement as ocalls. Others, like dlsym
I'm not sure about, as this is to obtain the address of a symbol in a shared library. Initially I thought this may have been because the address sanitizer was being linked dynamically, but this indicates that clang defaults to linking the address sanitizer statically. Indeed, when I link with -fsanitize=address -static-libsan
the result is unchanged.
My impression is that you cannot use ASAN as-is in the enclave environment because of the incompatibility of the run-time library, unless you manually port the run-time library by yourself.
Also, this document mentions that you should use clang (built-in) linker instead of ld for the final linking. Not sure if this is relevant.
Thanks @mingweishih.
The full link command output by our enclave's Makefile:
clang++-10 -g -nostdinc -m64 -fPIE -ftls-model=local-exec -fvisibility=hidden -fstack-protector-strong -fno-omit-fram
e-pointer -ffunction-sections -fdata-sections -mllvm -x86-speculative-load-hardening -Wa,-mlfence-before-indirect-branch=register -Wa,-m
lfence-before-ret=not -no-integrated-as -I/opt/openenclave/share/pkgconfig/../../include/openenclave/3rdparty/libcxx -I/opt/openenclave/
share/pkgconfig/../../include/openenclave/3rdparty/libc -I/opt/openenclave/share/pkgconfig/../../include/openenclave/3rdparty -I/opt/ope
nenclave/share/pkgconfig/../../include -DSGX_LEVEL_HW -o build/mu_enclave.so /opt/openenclave/share/pkgconfig/../../lib/
openenclave/enclave/objects-Debug/oeseal_gcmaes/seal_gcmaes.c.o -L/opt/openenclave/share/pkgconfig/../../lib/openenclave/enclave -nostdlib
-nodefaultlibs -nostartfiles -Wl,--no-undefined -Wl,-Bstatic -Wl,
-Bsymbolic -Wl,--export-dynamic -Wl,-pie -Wl,--build-id -Wl,-z,noexecstack -Wl,-z,now -Wl,-gc-sections -loeenclave-lvi-cfg -link-lvi-mit
igation -loelibcxx-lvi-cfg -loelibc-lvi-cfg -loesyscall-lvi-cfg -loecore-lvi-cfg -loecryptoopenssl-lvi-cfg -lopensslssl-lvi-cfg -lopenss
lcrypto-lvi-cfg -loelibc-lvi-cfg -loesyscall-lvi-cfg -loecore-lvi-cfg -L/usr/local/lib
x_oe -loelibc-lvi-cfg -loehostsock -loehostresolver -fsanitize=address -static-libsan
So we are using clang to link, and passing -v
to clang++ -g
reveals the ld
command that it's executing:
/usr/local/lvi-mitigation/bin/ld" -z relro --hash-style=gnu --build-id --eh-frame-hdr -m elf_x86_64 -dynamic-linker $
lib64/ld-linux-x86-64.so.2 -o build/my_enclave.so -L/opt/openenclave/share/pkgconfig/../../lib/openenclave/enclave -L/u
r/local/lib -L/usr/lib/gcc/x86_64-linux-gnu/7.5.0 -L/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../x86_64-linux-gnu -L/lib/x86_64-linux-g
u -L/lib/../lib64 -L/usr/lib/x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../.. -L/usr/lib/llvm-10/bin/../lib -L/lib -L/us
/lib --whole-archive /usr/lib/llvm-10/lib/clang/10.0.1/lib/linux/libclang_rt.asan-x86_64.a --no-whole-archive --dynamic-list=/usr/lib/l
vm-10/lib/clang/10.0.1/lib/linux/libclang_rt.asan-x86_64.a.syms --whole-archive /usr/lib/llvm-10/lib/clang/10.0.1/lib/linux/libclang_rt
asan_cxx-x86_64.a --no-whole-archive --dynamic-list=/usr/lib/llvm-10/lib/clang/10.0.1/lib/linux/libclang_rt.asan_cxx-x86_64.a.syms build/my_enclave.o
/opt/openenclave/share/pkgconfig/../../lib/openenclave/enclave/objects-Debug/oeseal_gcmaes/seal_gcmaes.c.o --
o-undefined -Bstatic -Bsymbolic --export-dynamic -pie --build-id -z noexecstack -z now -gc-sections -loeenclave-lvi-cfg -loelibcxx-lvi-
fg -loelibc-lvi-cfg -loesyscall-lvi-cfg -loecore-lvi-cfg -loecryptoopenssl-lvi-cfg -lopensslssl-lvi-cfg -lopensslcrypto-lvi-cfg -loelib
-lvi-cfg -loesyscall-lvi-cfg -loecore-lvi-cfg -loelibc-lvi-cfg -l
oehostsock -loehostresolver
And then it fails with: clang: error: linker command failed with exit code 1 (use -v to see invocation)
, so it appears clang is just calling ld.
I will try to port libasan. In general, techniques for introducing memory safety into a C++ SGX enclave are critical for security. A buffer overflow could easily leak secrets, for example. Address sanitizer is one possible approach to this, but we're open to other ideas as well.
And then it fails with:
clang: error: linker command failed with exit code 1 (use -v to see invocation)
, so it appears clang is just calling ld.
If you build with the LVI mitigation toolchain, the ld
will be used. But I guess the reason of using clang built-in linker is for linking the ASAN run-time library.
I will try to port libasan. In general, techniques for introducing memory safety into a C++ SGX enclave are critical for security. A buffer overflow could easily leak secrets, for example. Address sanitizer is one possible approach to this, but we're open to other ideas as well.
We have some previous work on enabling ASAN with customized LLVM toolchain: https://github.com/openenclave/openenclave-security/releases
Feel free to checkout it out and give it a try. Just note that there is not much documentation and we don't actively maintain the repo now.
In the paper I linked in my first post I see they had to do some work to port libasan to SGX:
I rebuilt and re-installed OpenEnclave without the LVI mitigated toolchain, such that no lvi binaries existed in the build environment, and re-built my project without any of the LVI flags. The result was the same. /usr/bin/ld
was called by clang.
Thanks very much for the link to openenclave-security, I will try it.
I installed the OpenEnclave LLVM 2.0 from here and installed it in the build environment at /usr
. Confirmed that the Makefile is using it:
clang version 8.0.1 openenclave-llvm
Target: x86_64-unknown-linux-gnu
However, it still fails with all the same missing symbols:
/usr/lib/clang/8.0.1/lib/linux/libclang_rt.asan-x86_64.a(interception_linux.cc.o): In function `__interception::GetRealFunctionAddress(char const*, unsigned long*, unsigned long, unsigned lon
/home/oe-llvm/compiler-rt/lib/interception/interception_linux.cc:33: undefined reference to `dlsym'
/home/oe-llvm/compiler-rt/lib/interception/interception_linux.cc:40: undefined reference to `dlsym'
/usr/lib/clang/8.0.1/lib/linux/libclang_rt.asan-x86_64.a(sanitizer_linux_libcdep.cc.o): In function `__sanitizer::GetThreadStackTopAndBottom(bool, unsigned long*, unsigned long*)':
/home/oe-llvm/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cc:142: undefined reference to `pthread_attr_init'
/home/oe-llvm/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cc:143: undefined reference to `pthread_getattr_np'
/home/oe-llvm/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cc:145: undefined reference to `pthread_attr_destroy'
/usr/lib/clang/8.0.1/lib/linux/libclang_rt.asan-x86_64.a(sanitizer_linux_libcdep.cc.o): In function `__sanitizer::InitTlsSize()':
/home/oe-llvm/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cc:245: undefined reference to `dlsym'
/usr/lib/clang/8.0.1/lib/linux/libclang_rt.asan-x86_64.a(sanitizer_linux_libcdep.cc.o): In function `__sanitizer::GetRSS()':
/home/oe-llvm/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cc:(.text._ZN11__sanitizer6GetRSSEv+0xe0): undefined reference to `getrusage'
/usr/lib/clang/8.0.1/lib/linux/libclang_rt.asan-x86_64.a(sanitizer_linux_libcdep.cc.o): In function `__sanitizer::WriteOneLineToSyslog(char const*)':
/home/oe-llvm/compiler-rt/lib/sanitizer_common/sanitizer_linux_libcdep.cc:762: undefined reference to `syslog'
/usr/lib/clang/8.0.1/lib/linux/libclang_rt.asan-x86_64.a(sanitizer_posix_libcdep.cc.o): In function `__sanitizer::ReleaseMemoryPagesToOS(unsigned long, unsigned long)':
/home/oe-llvm/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cc:68: undefined reference to `madvise'
/usr/lib/clang/8.0.1/lib/linux/libclang_rt.asan-x86_64.a(sanitizer_posix_libcdep.cc.o): In function `__sanitizer::NoHugePagesInRegion(unsigned long, unsigned long)':
/home/oe-llvm/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cc:74: undefined reference to `madvise'
/usr/lib/clang/8.0.1/lib/linux/libclang_rt.asan-x86_64.a(sanitizer_posix_libcdep.cc.o): In function `__sanitizer::DontDumpShadowMemory(unsigned long, unsigned long)':
/home/oe-llvm/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cc:82: undefined reference to `madvise'
/usr/lib/clang/8.0.1/lib/linux/libclang_rt.asan-x86_64.a(sanitizer_posix_libcdep.cc.o): In function `__sanitizer::setlim(int, unsigned long)':
/home/oe-llvm/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cc:103: undefined reference to `setrlimit'
/usr/lib/clang/8.0.1/lib/linux/libclang_rt.asan-x86_64.a(sanitizer_posix_libcdep.cc.o): In function `__sanitizer::SupportsColoredOutput(int)':
/home/oe-llvm/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cc:171: undefined reference to `isatty'
/usr/lib/clang/8.0.1/lib/linux/libclang_rt.asan-x86_64.a(sanitizer_posix_libcdep.cc.o): In function `__sanitizer::SetAlternateSignalStack()':
/home/oe-llvm/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cc:180: undefined reference to `sigaltstack'
/home/oe-llvm/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cc:191: undefined reference to `sigaltstack'
/usr/lib/clang/8.0.1/lib/linux/libclang_rt.asan-x86_64.a(sanitizer_posix_libcdep.cc.o): In function `__sanitizer::UnsetAlternateSignalStack()':
/home/oe-llvm/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cc:199: undefined reference to `sigaltstack'
/usr/lib/clang/8.0.1/lib/linux/libclang_rt.asan-x86_64.a(sanitizer_posix_libcdep.cc.o): In function `__sanitizer::IsAccessibleMemoryRange(unsigned long, unsigned long)':
/home/oe-llvm/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cc:286: undefined reference to `pipe'
/usr/lib/clang/8.0.1/lib/linux/libclang_rt.asan-x86_64.a(sanitizer_posix_libcdep.cc.o): In function `__sanitizer::GetNamedMappingFd(char const*, unsigned long)':
/home/oe-llvm/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cc:323: undefined reference to `shm_open'
/home/oe-llvm/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cc:327: undefined reference to `shm_unlink'
/usr/lib/clang/8.0.1/lib/linux/libclang_rt.asan-x86_64.a(sanitizer_posix_libcdep.cc.o): In function `__sanitizer::StartSubprocess(char const*, char const* const*, int, int, int)':
/home/oe-llvm/compiler-rt/lib/sanitizer_common/sanitizer_posix_libcdep.cc:487: undefined reference to `execv'
/usr/lib/clang/8.0.1/lib/linux/libclang_rt.asan-x86_64.a(sanitizer_symbolizer_libcdep.cc.o): In function `_M_construct<const char *>':
/usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/basic_string.tcc:219: undefined reference to `std::__cxx11::basic_string<char, std::char_traits<char>, std::alloca
reate(unsigned long&, unsigned long)'
/usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/basic_string.tcc:212: undefined reference to `std::__throw_logic_error(char const*)'
/usr/lib/clang/8.0.1/lib/linux/libclang_rt.asan-x86_64.a(sanitizer_symbolizer_libcdep.cc.o): In function `operator--':
/usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/stl_tree.h:302: undefined reference to `std::_Rb_tree_decrement(std::_Rb_tree_node_base*)'
/usr/lib/clang/8.0.1/lib/linux/libclang_rt.asan-x86_64.a(sanitizer_symbolizer_libcdep.cc.o): In function `_M_insert_<std::pair<const unsigned long long, std::__cxx11::basic_string<char> >, std
ed long long, std::pair<const unsigned long long, std::__cxx11::basic_string<char> >, std::_Select1st<std::pair<const unsigned long long, std::__cxx11::basic_string<char> > >, std::less<unsign
d::allocator<std::pair<const unsigned long long, std::__cxx11::basic_string<char> > > >::_Alloc_node>':
/usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/stl_tree.h:1755: undefined reference to `std::_Rb_tree_insert_and_rebalance(bool, std::_Rb_tree_node_base*, std::_
*, std::_Rb_tree_node_base&)'
/usr/lib/clang/8.0.1/lib/linux/libclang_rt.asan-x86_64.a(sanitizer_symbolizer_posix_libcdep.cc.o): In function `CreateTwoHighNumberedPipes':
/home/oe-llvm/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_posix_libcdep.cc:117: undefined reference to `pipe'
/home/oe-llvm/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_posix_libcdep.cc:117: undefined reference to `pipe'
/home/oe-llvm/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_posix_libcdep.cc:117: undefined reference to `pipe'
/home/oe-llvm/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_posix_libcdep.cc:117: undefined reference to `pipe'
/home/oe-llvm/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_posix_libcdep.cc:117: undefined reference to `pipe'
/usr/lib/clang/8.0.1/lib/linux/libclang_rt.asan-x86_64.a(sanitizer_symbolizer_posix_libcdep.cc.o): In function `InitializeSwiftDemangler':
/home/oe-llvm/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_posix_libcdep.cc:79: undefined reference to `dlsym'
/home/oe-llvm/compiler-rt/lib/sanitizer_common/sanitizer_symbolizer_posix_libcdep.cc:80: undefined reference to `dlerror'
/usr/lib/clang/8.0.1/lib/linux/libclang_rt.asan-x86_64.a(lsan_common_linux.cc.o): In function `IsLinker':
/home/oe-llvm/compiler-rt/lib/lsan/lsan_common_linux.cc:36: undefined reference to `getauxval'
Clang still calls ld, and it does appear to be linking against the asan* libraries that are within the /usr/lib/clang/8.0.1/
directories installed from OpenEnclave LLVM 2.0:
"/usr/bin/ld" -z relro --hash-style=gnu --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o build/my_enclave.so -L/opt/openenclave/share/pkgconfig/../.
./lib/openenclave/enclave -L/usr/local/lib -L/usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0 -L/usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0/../../../x86_64-linux-gnu -L/usr/bin/../lib/x86_64-linux-gnu -L/lib/x86_64-l
inux-gnu -L/lib/../lib64 -L/usr/lib/x86_64-linux-gnu -L/usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0/../../.. -L/usr/bin/../lib -L/lib -L/usr/lib --whole-archive /usr/lib/clang/8.0.1/lib/linux/libclang_rt.asan-x8
6_64.a --no-whole-archive --dynamic-list=/usr/lib/clang/8.0.1/lib/linux/libclang_rt.asan-x86_64.a.syms --whole-archive /usr/lib/clang/8.0.1/lib/linux/libclang_rt.asan_cxx-x86_64.a --no-whole-archive --dynamic-
list=/usr/lib/clang/8.0.1/lib/linux/libclang_rt.asan_cxx-x86_64.a.syms build/Enclave/Enclave_t.o /opt/openenclave/share/pkgconfig/../../lib/openenclave/enclave/objects-Debug/oeseal_gcmaes/seal_gcmaes.c.o --no-undefined -Bstatic -Bsymbolic --export-dynamic -pie --$
uild-id -z noexecstack -z now -gc-sections -loeenclave -loelibcxx -loelibc -loesyscall -loecore -loecryptoopenssl -lopensslssl -lopensslcrypto -loelibc -loesyscall -loecore -oe -leventsgx_oe -leent_opensslsgx_oe -loelibc -loehostsock -loehostresolver
Perhaps the OpenEnclave LLVM has a different usage for turning on the address sanitizer?
I was able to produce some AddressSanitizer functionality with openenclave-security's sample project.
Changing the lines in the build.sh to use the LLVM 2.0 rather than 1.0, and then changing this to #include <string>
rather than #include <string.h>
, I was able to build the sample project.
And the example heap overflow successfully triggers an AddressSanitizer crash and stack trace:
==0==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x7f4c44131030 at pc 0x7f4c40107ac6 bp 0x7f4c4803d3c0 sp 0x7f4c4803d3b8
WRITE of size 1 at 0x7f4c44131030 thread T16777215
==29460==WARNING: invalid path to external symbolizer!
==29460==WARNING: Failed to use and restart external symbolizer!
terminate called after throwing an instance of 'std::logic_error'
what(): basic_string::_M_construct null not valid
==29460== ERROR: libFuzzer: deadly signal
#0 0x559ab7 (/dev/stuff/openenclave-security/build/fuzzing_build/output/bin/enclavefuzz_host+0x559ab7)
#1 0x4765a8 (/dev/stuff/openenclave-security/build/fuzzing_build/output/bin/enclavefuzz_host+0x4765a8)
#2 0x457648 (/dev/stuff/openenclave-security/build/fuzzing_build/output/bin/enclavefuzz_host+0x457648)
#3 0x45760f (/dev/stuff/openenclave-security/build/fuzzing_build/output/bin/enclavefuzz_host+0x45760f)
#4 0x7f4c5a9de97f (/lib/x86_64-linux-gnu/libpthread.so.0+0x1297f)
#5 0x7f4c59e5be86 (/lib/x86_64-linux-gnu/libc.so.6+0x3ee86)
#6 0x7f4c59e5d7f0 (/lib/x86_64-linux-gnu/libc.so.6+0x407f0)
#7 0x7f4c5b3570a8 (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0x9d0a8)
#8 0x7f4c5b362505 (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xa8505)
#9 0x7f4c5b362570 (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xa8570)
#10 0x7f4c5b3627f4 (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xa87f4)
#11 0x7f4c5b3597dc (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0x9f7dc)
#12 0x572971 (/dev/stuff/openenclave-security/build/fuzzing_build/output/bin/enclavefuzz_host+0x572971)
#13 0x5d9583 (/dev/stuff/openenclave-security/build/fuzzing_build/output/bin/enclavefuzz_host+0x5d9583)
#14 0x5d5b64 (/dev/stuff/openenclave-security/build/fuzzing_build/output/bin/enclavefuzz_host+0x5d5b64)
#15 0x836a63 (/dev/stuff/openenclave-security/build/fuzzing_build/output/bin/enclavefuzz_host+0x836a63)
#16 0x8387f5 (/dev/stuff/openenclave-security/build/fuzzing_build/output/bin/enclavefuzz_host+0x8387f5)
#17 0x83744c (/dev/stuff/openenclave-security/build/fuzzing_build/output/bin/enclavefuzz_host+0x83744c)
#18 0x84d60d (/dev/stuff/openenclave-security/build/fuzzing_build/output/bin/enclavefuzz_host+0x84d60d)
#19 0x7f4c406005ef (/dev/sgx/enclave+0x6005ef)
Applying the -fsanitize=enclaveaddress
and -L../openenclave-security/build/fuzzing_build/output/lib -loefuzzsup_enc
to my project is not quite enough to reproduce this success in my project. There are multiple definitions of oe_ecalls_table
and oe_ecalls_table_size
, as well as some missing symbols getpwnam_r
, dprintf
, and others. I'm still working on all the necessary configurations. It may require building and linking against the openenclave port within openenclave-security rather than using openenclave main.
In an attempt to debug memory safety issues in our C++ SGX application, I've attempted to enable the address sanitizer when building our SGX application. I add
-fsanitize=address
to both the compile and link flags of my enclave build. Although compilation succeeds, the linker fails to find certain symbols:Is this expected? Has anyone successfully used the address sanitizer on an OpenEnclave SGX application?
There is precedent for running SGX applications with the address sanitizer, overview here.