Question about the ues of the library NTL

[DylanWangWQF]

Hi, @anakrish . Now, I'm trying to perform some encryptions and decryptions inside the enclave. In doing so, I need to build the static library GMP and NTL inside the enclave.

But as mentioned in issue openenclave-#3880, it's very difficult to do this. Currently, do we have any updates on solving this issue?

BTW, the library NTL is up to 43M which will take up a lot of enclave memory (128M). In fact, I only need part of the files in the NTL. Can I extract some files from the NTL, and then build NTL inside the enclave?

[anakrish]

Hi @DylanWangWQF,

I am discussing with the OE SDK team about the best way to publish apkman which will make it easier to use prebuilt-libraries within enclaves, as well as built libraries from source.

Since you've built NTL as a static library, the linker will pull in only those portions of it that is actually used by your enclave. Can you check the following

• What is the size of the enclave?
• What is the size of the enclave after running the strip command?

[DylanWangWQF]

@anakrish Many thanks for your help!

My question may sound a bit dumb, I'm new in SGX programming.

1. It seems that NTL is not available in alpinelinux.
2. Can I build the HElib and then link the helib.a ? In doing so, perform some encryptions and decryptions inside the enclave. But we should also build against OE's C++ library and then copy over helib.a and libstdc++.a from Alpine?

Sry, I'm working on implementing my paper experiment ASAP.

[anakrish]

@DylanWangWQF

You can build NTL, HELib from source on Alpine and then copy over the static libraries and then use them in your enclave. Yes, you'd need to copy over libstdc++.a from Alpine too.

For example, here is an use of boost and libstdc++ from Alpine: https://github.com/openenclave/openenclave/blob/3d6674a451608ea369f617fdd60f93cb5acdced2/tests/tools/apkman/cpp/boost/enc/CMakeLists.txt#L46-L52

Note:

• If the libraries use libc function not currently supported by OE, you'd have to implement them.

[DylanWangWQF]

Sry for proposing another question, @anakrish

This question is about: How to load Ciphertext, public key and secret key inside the enclave

In HElib, we can save the Contenxt, Public Key, Secret Key and Ciphertext to StringStream, Json or File. And then read them from the corresponding format.

StringStream

std::stringstream ss;
meta.data->context.writeTo(ss);   // publicKey.writeTo(ss)
helib::Context newContext = helib::Context::readFrom(ss);

JSON

std::stringstream ss;
meta.data->context.writeToJSON(ss);   
helib::Context newContext = helib::Context::readFromJSON(ss);

File

std::string path = pathPrefix + (pkNotSk ? ".pk" : ".sk");  
std::ofstream keysFile(path, std::ios::binary);
// write the context  
context.writeTo(keysFile);

// write the keys  
if (pkNotSk) {    
       const helib::PubKey& pk = secretKey;   
       pk.writeTo(keysFile); 
} else {
       secretKey.writeTo(keysFile);  
}

Before I perform encryptions or decryptions inside the enclave, I need to load Ciphertext, public key and secret key inside the enclave. But for loading content from the file, fstream is not supported in current OE https://github.com/openenclave/openenclave/issues/4005. While it seems that we can read the file inside the enclave as shown in openenclave/tests/syscall/dup/enc/enc.c and openenclave/tests/crypto/enclave/enc/enc.c, but no Docs to introduce this in OE. Or I have to load them through String (StringStream)? Or send the pointer about the Ciphertext to the enclave? Could you give me some suggestions? Thank you very much!!

[anakrish]

It would be better to send the cipher text to the enclave in a buffer, then create a string stream from the buffer and then perform subsequent operations.

Sending pointer to Ciphertext is not recommended since the host can change the contents of the ciphertext while the enclave is reading it.

You could declare an ecall in edl like this to ensure that the buffer is copied safely to the enclave: public void decrypt([in, count=length] uint8_t cipherText, uint64_t length);

[anakrish]

For using stringstream you might need to fix OE's locale implementation as done here: https://github.com/openenclave/openenclave/blob/3d6674a451608ea369f617fdd60f93cb5acdced2/tests/tools/apkman/cpp/boost/enc/enc.cpp#L18-L29

You could try without it and if needed add the above fix.

[DylanWangWQF]

Hi, @anakrish . Thanks for your help! It's really helpful! Recently, I found an extension of openenclave, sgx-lkl. Based on runtime sgx-lkl, it supports work on the unmodified library (sgx-lkl is responsible for building upon MUSL library).

So for my project, work on sgx-lkl or openenclave, the only difference is that I don't need to manually build the library on Alpine Linux? How about those functions which are currently not supported in OE?

[anakrish]

Hi @DylanWangWQF,

There are a couple of lift-and-shift solutions based on OpenEnclave

• SGX-LKL: https://github.com/lsds/sgx-lkl This runs a significant portion of the Linux kernel within the enclave. It is GLIBC based.
• Mystikos: https://github.com/deislabs/mystikos This runs a light-weight library OS within the enclave. It is currently MUSL based; but there is on going work to support GLIBC as well.

If you are looking for a lift-and-shift solution based on openenclave, I'd recommend Mystikos since it is much more light-weight compared to SGX-LKL and is also being actively maintained. As I understand it, currently for Mystikos you'd need to develop your application in an Alpine-Linux container if the application is complex.

[anakrish]

@DylanWangWQF you can follow up with @vtikoo for questions about Mystikos

[anakrish]

@DylanWangWQF If you plan to use apkman, then the command apkman exec sh will give you an alpine environment for building your libraries. On you build machine you can do (roughly)

git clone library-you-want-to-build
cd library-you-want-to-build
apkman exec sh
apk add build-base # compilers etc
apk add prerequisites-for-building-library
make

[DylanWangWQF]

Hi, @anakrish Many thanks for your help! I'm using apkman with OE. Before this, I built NTL, GMP and helib on VM with Alpine linux. Then copy libhelib.a,libgmp.a and libntl.ato /home/dylan/.apkman/alpine-fs/usr/lib and /home/dylan/.apkman/alpine-fs/usr/include.

dylan@dylan-OptiPlex-7070:~/.apkman/alpine-fs/usr/lib$ ls | grep a
bash
libgmp.a
libgmpxx.a
libhelib.a
libntl.a

dylan@dylan-OptiPlex-7070:~/.apkman/alpine-fs/usr/include$ ls
gmp.h  gmpxx.h  helib  NTL

But in the CmakeLists.txt, cannot find -L${APKMAN_ROOT}/usr/lib/gcc/x86_64-alpine-linux-musl/10.2.1. Here is my simple HE test: heenclave/enclave/CMakeLists.txt

target_link_libraries(
    enclave
    -nostdlib
    # Add library path and libraries.
    -L${APKMAN_ROOT}/usr/lib
    libhelib.a
    libgmp.a
    libntl.a
    libstdc++.a
    # for __umodti3
    -L${APKMAN_ROOT}/usr/lib/gcc/x86_64-alpine-linux-musl/10.2.1
    libgcc.a
    oelibcxx # For libunwind which has been patched for enclaves
    oelibc
    # Use apkman's ld to avoid bug with Ubuntu 18.04's binutils.
    -fuse-ld=${CMAKE_CURRENT_SOURCE_DIR}/ld
    # Emit warnings for unresolved symbols.
    -Wl,--warn-unresolved-symbols)

dylan@dylan-OptiPlex-7070:~/.apkman/alpine-fs/usr/lib$ ls
bash              libformw.so.6.2  libgmp.so.10      libgmpxx.so.4      libmenuw.so.6.2     libpanelw.so.6       libreadline.so.8    libstdc++.so.6.0.28         pkgconfig
engines-1.1       libgcc_s.so.1    libgmp.so.10.4.1  libgmpxx.so.4.6.1  libncursesw.so.6    libpanelw.so.6.2     libreadline.so.8.1  libtls-standalone.so.1
libcrypto.so.1.1  libgmp.a         libgmpxx.a        libhelib.a         libncursesw.so.6.2  libpkgconf.so.3      libssl.so.1.1       libtls-standalone.so.1.0.0
libformw.so.6     libgmp.so        libgmpxx.so       libmenuw.so.6      libntl.a            libpkgconf.so.3.0.0  libstdc++.so.6      modules-load.d

[anakrish]

@DylanWangWQF In my tests, APKMAN_ROOT is a cmake variable which is initialized with the output of the command apkman root. The following code is what ensures that apkman is initialized for use:

https://github.com/openenclave/openenclave/blob/3d6674a451608ea369f617fdd60f93cb5acdced2/tests/tools/apkman/CMakeLists.txt#L17-L33

[anakrish]

It looks like you need to install the gcc package that will bring in libgcc.a You can search for package using filenames here: https://pkgs.alpinelinux.org/contents?file=libgcc.a&path=&name=&branch=v3.13&arch=x86_64

The command apkman add build-base should setup apkman with build tools and libraries including libgcc.a.

[anakrish]

@DylanWangWQF As an example, here is a script that builds NTL using apkman.

#!/bin/sh

# Initialize apkman
apkman init

# Install packages apkman for building libraries
apkman add build-base cmake

# Install packages for building ntl
apkman add gmp gmp-dev perl

# Fetch sources
git clone https://github.com/libntl/ntl

# Build ntl
cd ntl/src

# Configure
apkman exec ./configure

# make
apkman exec make -j 8

# Display library
find . -name *.a

The command apkman exec cmd executes the specified cmd in apkman's Alpine Linux environment. E.g: apkman exec sh will give you a complete alpine linux prompt. apkman exec ./configure will run the configure script in the current folder mapped to the alpine linux environment.

One of the goals of apkman is that you can seamlessly build libraries without having to use a separate Alpine Linux VM.

Note: I'm not that familiary with NTL/HElib etc.

[DylanWangWQF]

Hi @anakrish . I have already successfully re-installed these static libraries through apkman shell (include HElib). It's really very convenient to build packages in the Alpine environment.

In my tests, APKMAN_ROOT is a cmake variable which is initialized with the output of the command apkman root. The following code is what ensures that apkman is initialized for use:

https://github.com/openenclave/openenclave/blob/3d6674a451608ea369f617fdd60f93cb5acdced2/tests/tools/apkman/CMakeLists.txt#L17-L33

Instead of setting APKMAN_ROOT as a cmake variable, can I use the absolute path in CMakeLists.txt after I manually install those libraries. e.g.,

     -L$~/.apkman/alpine-fs/usr/lib

    libhelib.a libgmp.a libntl.a libstdc++.a
    # for __umodti3

    -L$~/.apkman/alpine-fs/usr/lib/gcc/x86_64-alpine-linux-musl/10.2.1

    libgcc.a oelibcxx  oelibc
    # Use apkman's ld to avoid bug with Ubuntu 18.04's binutils.
    -fuse-ld=${CMAKE_CURRENT_SOURCE_DIR}/ld

And we can remove this:

add_custom_target(
        gmp
        COMMAND apkman add gmp-dev
        DEPENDS apkman-init)

[anakrish]

@DylanWangWQF Yes, hard-coding the paths should also work. With the obvious drawback that the path is hard-coded :)

If you have built debug versions of the libraries, then you will be able to step into them via oegdb.

[DylanWangWQF]

Hi @anakrish . I have deleted those redundant results and questions. And here I summarize some notes to notice others who may use apkman and list my questions.

1. It would be better to set APKMAN_ROOT as a cmake variable, use ${APKMAN_ROOT}/ in the paths in CMakeLists.txt.
2. If we code according to OE Samples, add openenclave::oelibcxx in target_link_libraries in enclave/CMakeLists.txt, otherwise it automatically searchs the header files in /usr/local/include. (Don't forget to source the location of the installed OpenEnclave).
3. Don't forget to link include directories (without installing them in enclave/CMakeLists.txt) ${APKMAN_ROOT}/usr/include.

Question 1: Now, I hit this issue when make the file

Scanning dependencies of target enclave
[ 18%] Building CXX object enclave/CMakeFiles/enclave.dir/ecalls.cpp.o
In file included from /home/dylan/mysamples/heenclave/enclave/dispatcher.h:7:0,
                 from /home/dylan/mysamples/heenclave/enclave/ecalls.cpp:4:
/opt/openenclave/include/openenclave/3rdparty/libcxx/cmath:314:9: error: ‘::signbit’ has not been declared
 using ::signbit;
         ^~~~~~~
/opt/openenclave/include/openenclave/3rdparty/libcxx/cmath:315:9: error: ‘::fpclassify’ has not been declared
 using ::fpclassify;
         ^~~~~~~~~~
/opt/openenclave/include/openenclave/3rdparty/libcxx/cmath:316:9: error: ‘::isfinite’ has not been declared
 using ::isfinite;
         ^~~~~~~~
/opt/openenclave/include/openenclave/3rdparty/libcxx/cmath:317:9: error: ‘::isinf’ has not been declared
 using ::isinf;
         ^~~~~
/opt/openenclave/include/openenclave/3rdparty/libcxx/cmath:318:9: error: ‘::isnan’ has not been declared
 using ::isnan;

etc.....

Current CmakeList:

target_include_directories(
  enclave
  PRIVATE ${CMAKE_CURRENT_BINARY_DIR}
                    ${CMAKE_SOURCE_DIR}
                    ${APKMAN_ROOT}/usr/include)

target_link_libraries(
    enclave 
    -nostdlib
    # Add library path and libraries.
    # -L$~/.apkman/alpine-fs/usr/lib
    -L${APKMAN_ROOT}/usr/lib
    libhelib.a
    libgmp.a
    libntl.a
    libstdc++.a
    # for __umodti3
    -L${APKMAN_ROOT}/usr/lib/gcc/x86_64-alpine-linux-musl/10.2.1
    libgcc.a
    openenclave::oeenclave
    openenclave::oecrypto${OE_CRYPTO_LIB}
    openenclave::oelibcxx
    openenclave::oelibc
    # Use apkman's ld to avoid bug with Ubuntu 18.04's binutils.
    -fuse-ld=${CMAKE_CURRENT_SOURCE_DIR}/ld
    # Emit warnings for unresolved symbols.
    -Wl,--warn-unresolved-symbols)

Question 2: How should I execute ld command? I got this error before:

c++: error: unrecognized command line option ‘-fuse-ld=/home/dylan/mysamples/heenclave/enclave/ld’

Many thanks for your help and patience!

[anakrish]

@DylanWangWQF

OE's libcxx is a bit out of date. Hence you see those compile errors.

The way I worked around such issues is to compile the cpp file using apkman exec and then include the generated object file in the enclave.

Here is an example of using C++20 coroutines: https://github.com/openenclave/openenclave/blob/3d6674a451608ea369f617fdd60f93cb5acdced2/tests/tools/apkman/cpp/cpp20/enc/CMakeLists.txt#L17-L31

The file is compiled using apkman and clang/gcc.

add_custom_command(
  OUTPUT main.o
  COMMAND apkman exec g++ -o main.o -g -Og -c -fcoroutines -std=c++20
          ${CMAKE_CURRENT_SOURCE_DIR}/main.cpp
  DEPENDS main.cpp apkman-init)

Then the output object is added to the enclave:

add_enclave(
  TARGET
  cpp20_enc
  UUID
  71b0822f-42a3-4543-a97c-ca491f76b82c
  SOURCES
  enc.cpp
  main.o  <---- generated via compiling using apkman
  ${CMAKE_CURRENT_BINARY_DIR}/test_t.c)

[anakrish]

@DylanWangWQF Alternatively, you can specify the apkman libcxx in the include path so that they are used instead of OE's libcxx:

https://github.com/openenclave/openenclave/blob/3d6674a451608ea369f617fdd60f93cb5acdced2/tests/tools/apkman/cpp/protobuf/enc/CMakeLists.txt#L36-L45

enclave_include_directories(
  protobuf_enc
  PRIVATE
  ${CMAKE_CURRENT_BINARY_DIR}
  # Protobuf headers available via protobuf-dev library.
  # C++
  ${APKMAN_ROOT}/usr/include/c++/10.2.1
  ${APKMAN_ROOT}/usr/include/c++/10.2.1/x86_64-alpine-linux-musl
  # C
  ${APKMAN_ROOT}/usr/include)

[anakrish]

You can try out either of the above two approaches and see which one works best overall.

[anakrish]

c++: error: unrecognized command line option ‘-fuse-ld=/home/dylan/mysamples/heenclave/enclave/ld’

You can try removing the -fuse-ld option and see if it works. The option is used in some apkman tests since Ubuntu 18.04's ld has a bug which prevents it from handling compressed debug sections in object files/archives.

In some apkman tests, -fuse-ld is used to specify the path to a script which invokes Alpine Linux's ld via apkman exec ld "$@"

[DylanWangWQF]

Hi, @anakrish . I specify the apkman libcxx in the include path, the issue about error: ‘::signbit’ has not been declared has been solved. Then I hit the previous issue, /usr/bin/ld: /home/dylan/.apkman/alpine-fs/usr/lib/libhelib.a(BenesNetwork.cpp.o): unable to initialize decompress status for section .debug_info, largely due to the Ubuntu 18.04's ld bug.

Scanning dependencies of target enclave
[ 18%] Building CXX object enclave/CMakeFiles/enclave.dir/ecalls.cpp.o
[ 27%] Building CXX object enclave/CMakeFiles/enclave.dir/dispatcher.cpp.o
[ 36%] Building C object enclave/CMakeFiles/enclave.dir/heenclave_t.c.o
[ 45%] Linking CXX executable enclave
/usr/bin/ld: /home/dylan/.apkman/alpine-fs/usr/lib/libhelib.a(BenesNetwork.cpp.o): unable to initialize decompress status for section .debug_info
/usr/bin/ld: /home/dylan/.apkman/alpine-fs/usr/lib/libhelib.a(BenesNetwork.cpp.o): unable to initialize decompress status for section .debug_info
/usr/bin/ld: /home/dylan/.apkman/alpine-fs/usr/lib/libhelib.a(Context.cpp.o): unable to initialize decompress status for section .debug_info
/usr/bin/ld: /home/dylan/.apkman/alpine-fs/usr/lib/libhelib.a(Context.cpp.o): unable to initialize decompress status for section .debug_info
/home/dylan/.apkman/alpine-fs/usr/lib/libhelib.a: unable to add symbols: unrecognized file format
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Update: After I add the command -fuse-ld=${CMAKE_CURRENT_SOURCE_DIR}/ld, the bug has been solved. But cannot find other OE libs.

[ 18%] Building CXX object enclave/CMakeFiles/enclave.dir/ecalls.cpp.o
[ 27%] Building CXX object enclave/CMakeFiles/enclave.dir/dispatcher.cpp.o
[ 36%] Building C object enclave/CMakeFiles/enclave.dir/heenclave_t.c.o
[ 45%] Linking CXX executable enclave
ld: cannot find /opt/openenclave/lib/openenclave/enclave/liboeenclave.a: No such file or directory
ld: cannot find /opt/openenclave/lib/openenclave/enclave/liboecryptombedtls.a: No such file or directory
ld: cannot find /opt/openenclave/lib/openenclave/enclave/libmbedtls.a: No such file or directory
ld: cannot find /opt/openenclave/lib/openenclave/enclave/libmbedx509.a: No such file or directory
ld: cannot find /opt/openenclave/lib/openenclave/enclave/libmbedcrypto.a: No such file or directory
ld: cannot find /opt/openenclave/lib/openenclave/enclave/liboelibc.a: No such file or directory
ld: cannot find /opt/openenclave/lib/openenclave/enclave/liboesyscall.a: No such file or directory
ld: cannot find /opt/openenclave/lib/openenclave/enclave/liboecore.a: No such file or directory

[anakrish]

@DylanWangWQF

apkman makes only /home accessible to apkman exec command. Hence, /opt is not available. Maybe based on your experience, apkman ought to map /opt too - especially since OE is installed by default to /opt.

As a workaround:

• You can tweak apkman to map /opt too. https://github.com/openenclave/openenclave/blob/3d6674a451608ea369f617fdd60f93cb5acdced2/tools/apkman/apkman#L176 Note, if apkman uses pshare, then you'd need to map /opt in the section above.
• (Or) Copy /opt/openenclave to ~/.apkman/alpinefs/opt/openenclave
• (Or) Copy the oe libs to some folder under the home directory and specify them explicitly.

[DylanWangWQF]

@anakrish

• (Or) Copy /opt/openenclave to ~/.apkman/alpinefs/opt/openenclave

I adopt the second method, while I get the clang error: Here is the error log: make.log

/home/dylan/GithubCode/ntl/src/lip.cpp:5667: warning: undefined reference to `__gmpz_clear'
ld: /home/dylan/GithubCode/ntl/src/lip.cpp:5667: warning: undefined reference to `__gmpz_clear'
ld: /home/dylan/GithubCode/ntl/src/lip.cpp:5667: warning: undefined reference to `__gmpz_clear'
ld: /home/dylan/GithubCode/ntl/src/lip.cpp:5667: warning: undefined reference to `__gmpz_clear'
ld: /home/dylan/GithubCode/ntl/src/lip.cpp:5667: warning: undefined reference to `__gmpz_clear'
ld: /home/dylan/.apkman/alpine-fs/usr/lib/libntl.a(lip.o):/home/dylan/GithubCode/ntl/src/lip.cpp:5667: warning: more undefined references to `__gmpz_clear' follow
ld: /home/dylan/.apkman/alpine-fs/usr/lib/libntl.a(GetTime.o): in function `_ntl_GetTime()':
/home/dylan/GithubCode/ntl/src/GetTime.cpp:13: warning: undefined reference to `getrusage'
ld: /home/dylan/.apkman/alpine-fs/usr/lib/libstdc++.a(thread.o): in function `std::thread::hardware_concurrency()':
/home/buildozer/aports/main/gcc/src/gcc-10.2.1_pre1/libstdc++-v3/src/c++11/thread.cc:177: warning: undefined reference to `get_nprocs'
ld: /home/dylan/.apkman/alpine-fs/usr/lib/libstdc++.a(time_members.o): in function `std::__timepunct<wchar_t>::_M_put(wchar_t*, unsigned long, wchar_t const*, tm const*) const':
/home/buildozer/aports/main/gcc/src/build/x86_64-alpine-linux-musl/libstdc++-v3/src/c++98/time_members.cc:136: warning: undefined reference to `wcsftime'
clang: error: linker command failed with exit code 1 (use -v to see invocation)

I'm not sure if the different ld cause this error, it's confusing

target_include_directories(
  enclave
  PRIVATE ${CMAKE_CURRENT_BINARY_DIR}
                    ${CMAKE_SOURCE_DIR}
                    # C++
                    ${APKMAN_ROOT}/usr/include/c++/10.2.1
                    ${APKMAN_ROOT}/usr/include/c++/10.2.1/x86_64-alpine-linux-musl
                    # C
                    ${APKMAN_ROOT}/usr/include)

target_link_libraries(
    enclave 
    -nostdlib
    # Add library path and libraries.
    # -L$~/.apkman/alpine-fs/usr/lib
    -L${APKMAN_ROOT}/usr/lib
    libhelib.a
    libgmp.a
    libntl.a
    libstdc++.a
    # for __umodti3
    -L${APKMAN_ROOT}/usr/lib/gcc/x86_64-alpine-linux-musl/10.2.1
    libgcc.a
    openenclave::oeenclave
    openenclave::oecrypto${OE_CRYPTO_LIB}
    openenclave::oelibcxx
    openenclave::oelibc
    # Use apkman's ld to avoid bug with Ubuntu 18.04's binutils.
    -fuse-ld=${CMAKE_CURRENT_SOURCE_DIR}/ld
    # Emit warnings for unresolved symbols.
    -Wl,--warn-unresolved-symbols)

[anakrish]

It's a library order issue. libgmp.a ought to come after libntl.a since NTL depends on GMP.

[DylanWangWQF]

@anakrish I checked it out just now....Really sry for this stupid question.

BTW, in the previous normal Ubuntu environment, I also link the math library -lm by g++ main.cpp -o TestHE -L/usr/local/lib ../src1/libhelib.a /usr/local/lib/libntl.a /usr/local/lib/libgmp.a -lm -std=c++17 -O2 -I/../src1/helib -pthread

But I checked it's empty in Alpine.

dylan@dylan-OptiPlex-7070:~/.apkman/alpine-fs/usr/lib$ ar -t libm.a

For the purpose of record: https://www.musl-libc.org/doc/1.0.0/manual.html

Why is libm.a empty? On musl, the entire standard library is included in a single library file — libc.a for static linking, and libc.so for dynamic linking. This significantly improves the efficiency of dynamic linking, and avoids all sorts of symbol interposition bugs that arise when you split the libraries up — bugs which have plagued glibc for more than a decade.

Why not just omit libm.a, libpthread.a, etc. entirely? POSIX says -lm, -lpthread, etc. are valid options to the compiler and conforming applications must use these options when they need the corresponding functionality, and the easiest way to comply with that requirement (and avoid breaking applications) is with empty .a files by those names.

[DylanWangWQF]

Hi @anakrish .
I directly comment on a function in HElib, remove std::localtime. In doing so, the warning about undefined reference localtime is removed. But for some other functions, e.g., __gmp_exception in libgmp.a and std::thread::hardware_concurrency() in libstdc++.a, do I need to comment in the same way? Since we use apkman add gmp-dev to install it the apkman path.

[ 45%] Linking CXX executable enclave
ld: /opt/openenclave/lib/openenclave/enclave/liboecore.a(threadlocal.c.o): in function `__cxa_thread_atexit':
/home/dylan/openenclave/enclave/core/sgx/threadlocal.c:379: multiple definition of `__cxa_thread_atexit'; /home/dylan/.apkman/alpine-fs/usr/lib/libstdc++.a(atexit_thread.o):/home/buildozer/aports/main/gcc/src/gcc-10.2.1_pre1/libstdc++-v3/libsupc++/atexit_thread.cc:121: first defined here

ld: /home/dylan/.apkman/alpine-fs/usr/lib/libntl.a(fileio.o): in function `NTL::UniqueID[abi:cxx11]()':
/home/dylan/GithubCode/ntl/src/fileio.cpp:119: warning: undefined reference to `clock'

ld: /home/dylan/.apkman/alpine-fs/usr/lib/libntl.a(GetTime.o): in function `_ntl_GetTime()':
/home/dylan/GithubCode/ntl/src/GetTime.cpp:13: warning: undefined reference to `getrusage'

ld: /home/dylan/.apkman/alpine-fs/usr/lib/libgmp.a(errno.o): in function `__gmp_exception':
errno.c:(.text+0x10): warning: undefined reference to `raise'

ld: /home/dylan/.apkman/alpine-fs/usr/lib/libstdc++.a(thread.o): in function `std::thread::hardware_concurrency()':
/home/buildozer/aports/main/gcc/src/gcc-10.2.1_pre1/libstdc++-v3/src/c++11/thread.cc:177: warning: undefined reference to `get_nprocs'

ld: /home/dylan/.apkman/alpine-fs/usr/lib/libstdc++.a(time_members.o): in function `std::__timepunct<wchar_t>::_M_put(wchar_t*, unsigned long, wchar_t const*, tm const*) const':
/home/buildozer/aports/main/gcc/src/build/x86_64-alpine-linux-musl/libstdc++-v3/src/c++98/time_members.cc:136: warning: undefined reference to `wcsftime'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
enclave/CMakeFiles/enclave.dir/build.make:133: recipe for target 'enclave/enclave' failed

Additionally, for the issue multiple definition of __cxa_thread_atexit, have you hit this issue? This is my CMakeLists.txt and OE location:

target_include_directories(
  enclave
  PRIVATE ${CMAKE_CURRENT_BINARY_DIR}
                    #${APKMAN_ROOT}/opt/openenclave/include
                    # C++
                    ${APKMAN_ROOT}/usr/include/c++/10.2.1
                    ${APKMAN_ROOT}/usr/include/c++/10.2.1/x86_64-alpine-linux-musl
                    # C
                    ${APKMAN_ROOT}/usr/include)

target_compile_options(
  enclave
  PRIVATE
  -Wno-shorten-64-to-32
  -Wno-bitwise-op-parentheses
  -Wno-shift-op-parentheses
  -Wno-sign-conversion
  -Wno-unused-parameter
  -Wno-implicit-int-conversion)

target_link_libraries(
    enclave 
    -nostdlib
    # Add library path and libraries.
    # -L$~/.apkman/alpine-fs/usr/lib
    -L${APKMAN_ROOT}/usr/lib
    libhelib.a
    libntl.a
    libgmp.a
    #libc.a
    libstdc++.a
    # for __umodti3
    -L${APKMAN_ROOT}/usr/lib/gcc/x86_64-alpine-linux-musl/10.2.1
    libgcc.a
    openenclave::oeenclave
    openenclave::oecrypto${OE_CRYPTO_LIB}
    openenclave::oelibcxx
    openenclave::oelibc
    # Use apkman's ld to avoid bug with Ubuntu 18.04's binutils.
    -fuse-ld=${CMAKE_CURRENT_SOURCE_DIR}/ld
    # Emit warnings for unresolved symbols.
    -Wl,--warn-unresolved-symbols)

dylan@dylan-OptiPlex-7070:/opt/openenclave$ ls
bin  include  lib  share

dylan@dylan-OptiPlex-7070:~/.apkman/alpine-fs/opt/openenclave$ ls
bin  include  lib  share

[anakrish]

The missing references are the functions that are not yet implemented in OE. You can either implement them or modify the code so that they are not used. Sometimes some of those missing functions are never called during execution and can therefore have empty implementations.

I have not seen the __cxa_thread_atexit error before. Ideally, you want OE's implementation to be picked up since it works within enclaves.

[DylanWangWQF]

Hi, @anakrish . Basically, I have to also edit the libstdc++ files (comment those functions) and rebuild them to libstdc++.a to avoid the below files, right? I just want to make sure this since it is a much more lengthy process for manually building libstdc++.a.

ld: /opt/openenclave/lib/openenclave/enclave/liboecore.a(threadlocal.c.o): in function `__cxa_thread_atexit':
/home/dylan/openenclave/enclave/core/sgx/threadlocal.c:379: multiple definition of `__cxa_thread_atexit'; /home/dylan/.apkman/alpine-fs/usr/lib/libstdc++.a(atexit_thread.o):/home/buildozer/aports/main/gcc/src/gcc-10.2.1_pre1/libstdc++-v3/libsupc++/atexit_thread.cc:121: first defined here

ld: /home/dylan/.apkman/alpine-fs/usr/lib/libstdc++.a(thread.o): in function `std::thread::hardware_concurrency()':
/home/buildozer/aports/main/gcc/src/gcc-10.2.1_pre1/libstdc++-v3/src/c++11/thread.cc:177: warning: undefined reference to `get_nprocs'

ld: /home/dylan/.apkman/alpine-fs/usr/lib/libstdc++.a(time_members.o): in function `std::__timepunct<wchar_t>::_M_put(wchar_t*, unsigned long, wchar_t const*, tm const*) const':
/home/buildozer/aports/main/gcc/src/build/x86_64-alpine-linux-musl/libstdc++-v3/src/c++98/time_members.cc:136: warning: undefined reference to `wcsftime'

https://github.com/openenclave/openenclave/blob/3d6674a451608ea369f617fdd60f93cb5acdced2/tests/tools/apkman/cpp/boost/enc/CMakeLists.txt#L47

[anakrish]

I'd recommend that you implement the undefined functions (e.g get_nprocs, wcsftime) instead of modifying libstdc++. Modifying libstdc++ is complicated and best avoided.

I'm not really sure what to do about the multiple definition error since I haven't seen it before and don't have reproduction steps. One of your libraries (NTL or helib) is using thread-locals and that is causing a conflict with OE's thread-local implementation.

Can you

• build NTL/helib without using thread-locals? There might be some way to configure them to avoid thread-locals.
• Or try passing the flag -ftls-model=local-exec or -ftls-model=initial-exec when building NTL/helib. These flags could result in code for thread-locals that is compatible with OE.

[DylanWangWQF]

Hi, @anakrish For these two methods, still hit the above issues (undefined reference to get_nprocs inlibstdc++.a(thread.o): is removed):

ld: /opt/openenclave/lib/openenclave/enclave/liboecore.a(threadlocal.c.o): in function `__cxa_thread_atexit':
/home/dylan/openenclave/enclave/core/sgx/threadlocal.c:379: multiple definition of `__cxa_thread_atexit'; /home/dylan/.apkman/alpine-fs/usr/lib/libstdc++.a(atexit_thread.o):/home/buildozer/aports/main/gcc/src/gcc-10.2.1_pre1/libstdc++-v3/libsupc++/atexit_thread.cc:121: first defined here

ld: /home/dylan/.apkman/alpine-fs/usr/lib/libstdc++.a(time_members.o): in function `std::__timepunct<wchar_t>::_M_put(wchar_t*, unsigned long, wchar_t const*, tm const*) const':
/home/buildozer/aports/main/gcc/src/build/x86_64-alpine-linux-musl/libstdc++-v3/src/c++98/time_members.cc:136: warning: undefined reference to `wcsftime'

For the second method, I rebuild with these commands:

//NTL
// CXXFLAGS="-ftls-model=local-exec"
apkman exec ./configure NTL_GMP_LIP=on SHARED=on GMP_PREFIX=/home/dylan/Downloads/gmp_install NTL_THREADS=off NTL_THREAD_BOOST=off PREFIX=/home/dylan/Downloads/libwithflags/ntl_flags CXXFLAGS="-ftls-model=local-exec"

//HElib
// -DCMAKE_CXX_FLAGS="-ftls-model=local-exec"
apkman exec cmake -DGMP_DIR="/home/dylan/Downloads/gmp_install" -DNTL_DIR="/home/dylan/Downloads/libwithflags/ntl_flags" DENABLE_THREADS=OFF -DCMAKE_CXX_FLAGS="-ftls-model=local-exec" -DCMAKE_INSTALL_PREFIX=/home/dylan/Downloads/libwithflags/helib_flags  ..

Are these commands correct?

[DylanWangWQF]

Some cpp files of HElib: HElib/src/CModulus.cpp HElib/src/DoubleCRT.cpp

HElib/src/Ctxt.cpp:
 #include <NTL/BasicThreadPool.h>
 #include <NTL/ZZ.h>

HElib/src/DoubleCRT.cpp:
#include <NTL/ZZVec.h>
#include <NTL/BasicThreadPool.h>
static thread_local NTL::Vec<long> tls_ivec;
  static thread_local NTL::Vec<long> tls_pvec;
  static thread_local NTL::Vec<NTL::Vec<long>> tls_remtab;
  static thread_local NTL::Vec<NTL::zz_pX> tls_tmpvec;

I even extract the .cpp files I used in the smaple code to compile by this command:

g++ -c -I/home/dylan/Downloads/libwithflags/ntl_flags1/include -I/home/dylan/Downloads/gmp_install/include -ftls-model=local-exec -std=c++17 *.cpp
ar rc libhelib.a *.o
rm *.o

[anakrish]

@DylanWangWQF, __cxa_thread_atexit is referenced whenever a C++ thread_local variable has a non trivial destructor. OE's implementation of __cxa_thread_atexit is defined in oecore which is generally the last archive in the linker command line.

To ensure that OE's implementation is picked up, a workaround is to specify (lib) oecore a second time, before libstdc++.

https://github.com/anakrish/openenclave/blob/1de7e64972faddb20979c73a32094c8fbbc347bf/tests/tools/apkman/cpp/thread_local/enc/enc.cpp#L8-L20

https://github.com/anakrish/openenclave/blob/1de7e64972faddb20979c73a32094c8fbbc347bf/tests/tools/apkman/cpp/thread_local/enc/CMakeLists.txt#L25-L41

[DylanWangWQF]

Hi, @anakrish . Finally, make is successful!! But when I make run, I got this error.

security level = 98.4643
Host: create enclave for image:/home/dylan/mysamples/heenclave/build/enclave/enclave.signed
Host: check context_len: 364
Host: check hecontext: |HE[
Host: send and receive Ctxt to enclave:
Enclave: /home/dylan/mysamples/heenclave/enclave/dispatcher.cpp(30): Enclave: receive the HE params
Enclave: /home/dylan/mysamples/heenclave/enclave/dispatcher.cpp(32): output the test value with 1

CMakeFiles/run.dir/build.make:57: recipe for target 'CMakeFiles/run' failed
make[3]: *** [CMakeFiles/run] Segmentation Fault  (core dumped)

Debug the sample in VScode, shown as below:

Breakpoint in host.cpp: 
result = oe_create_heenclave_enclave(argv[1], OE_ENCLAVE_TYPE_SGX, flags, NULL, 0, &enclave);

Breakpoint 2, main (argc=2, argv=0x7fffffffdbf8) at /home/dylan/mysamples/heenclave/host/host.cpp:92
92      const uint32_t flags = OE_ENCLAVE_FLAG_DEBUG;
Execute debugger commands using "-exec <command>", for example "-exec info registers" will list registers in use (when GDB is the debugger)
oegdb: Symbols loaded for enclave 

=memory-changed,thread-group="i1",addr="0x00007ffff3641008",len="0x4"
=memory-changed,thread-group="i1",addr="0x00007ffff3a49008",len="0x4"
oegdb: All tcs set to debug for enclave 

Program received signal SIGSEGV, Segmentation fault.
std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::length (this=0x0) at /home/buildozer/aports/main/gcc/src/build/x86_64-alpine-linux-musl/libstdc++-v3/include/bits/basic_string.h:907

Unable to open 'basic_string.h': Unable to read file '/home/buildozer/aports/main/gcc/src/build/x86_64-alpine-linux-musl/libstdc++-v3/include/bits/basic_string.h' (Error: Unable to resolve non-existing file '/home/buildozer/aports/main/gcc/src/build/x86_64-alpine-linux-musl/libstdc++-v3/include/bits/basic_string.h').

For using stringstream you might need to fix OE's locale implementation as done here: https://github.com/openenclave/openenclave/blob/3d6674a451608ea369f617fdd60f93cb5acdced2/tests/tools/apkman/cpp/boost/enc/enc.cpp#L18-L29

You could try without it and if needed add the above fix.

You mentioned the issue fix OE's locale implementation before. I don't know if this is the reason, but when I add this part in ecalls.cpp, got multiple definition of setlocale:

ld: /opt/openenclave/lib/openenclave/enclave/liboelibc.a(locale.c.o): in function `setlocale':
/home/dylan/openenclave/libc/locale.c:58: multiple definition of `setlocale'; CMakeFiles/enclave.dir/ecalls.cpp.o:ecalls.cpp:(.text.setlocale+0x0): first defined here

[DylanWangWQF]

@anakrish When I comment the invoked HElib function inside the enclave, the target runs successfully. Code

int ecall_dispatcher::CtxtTransform(
    const char* hecontext,
    size_t context_len)
{
    std::stringstream ess;

    TRACE_ENCLAVE("Enclave: receive the HE params");
    int test =1;
    TRACE_ENCLAVE("output the test value with %d", test);
    std::cout << "the passed string length: " << context_len << std::endl;
    std::cout << "the passed char pointer: " << hecontext << std::endl;

    ess.clear();
    std::string e_hecontext = "test";
    std::cout << e_hecontext << std::endl;
    e_hecontext = std::string(hecontext, hecontext + context_len);
    std::cout << "Get the string: " << e_hecontext << std::endl;
    ess << e_hecontext;
    //helib::Context e_context = helib::Context::readFrom(ess);
    ess.str("");

    TRACE_ENCLAVE("Enclave: print the context: ");
    //e_context.printout();

    // invoke the NTL library to generate a matrix:
    std::cout << "Enclave: randomly generate a matrix" << std::endl;
    NTL::Mat<NTL::ZZ> Amat;
    Amat.SetDims(16, 16);

    for(size_t i = 0; i < 16 ; i++){
        for(size_t j = 0; j < 16; j++){
            Amat[i][j]= NTL::to_ZZ(i + 1);
        }
    }
    std::cout << "Enclave: print the generated matrix: " << Amat << std::endl;
}

Result

Host: create enclave for image:/home/dylan/mysamples/heenclave/build/enclave/enclave.signed
Host: check context_len: 364
Host: check hecontext: |HE[
Host: send and receive Ctxt to enclave:
Enclave: /home/dylan/mysamples/heenclave/enclave/dispatcher.cpp(30): Enclave: receive the HE params
Enclave: /home/dylan/mysamples/heenclave/enclave/dispatcher.cpp(32): output the test value with 1
the passed string length: 364
the passed char pointer: |HE[
test
Get the string: |HE[
]HE||CN[
2+��������    @$@

       
�
��
��
�
(<
��

�
�
'
]CN|
Enclave: /home/dylan/mysamples/heenclave/enclave/dispatcher.cpp(46): Enclave: print the context: 
Enclave: randomly generate a matrix
Enclave: print the generated matrix:
[[1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1]
[2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2]
[3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3]
[4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4]
[5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5]
[6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6]
[7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7]
[8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8]
[9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9]
[10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10]
[11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11]
[12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12]
[13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13]
[14 14 14 14 14 14 14 14 14 14 14 14 14 14 14 14]
[15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15]
[16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16]
]
Host: terminate the enclave
Host: done  succeeded
[100%] Built target run

We can see that all the variables are printed and successfully invoke the functions in NTL to genereate a matrix. But when I invoke the helib function readfrom in the last comment, hit the issue Segmentation Fault (core dumped).

Run /opt/openenclave/bin/oegdb -arg ./host/helloworld_host ./enc/helloworld_enc.signed:

Get the string: |HE[
]HE||CN[
2+��������    @$@

       
�
��
��
�
(<
��

�
�
'
]CN|

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff23276fe in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::length() const ()
    at /home/buildozer/aports/main/gcc/src/gcc-10.2.1_pre1/libstdc++-v3/libsupc++/eh_alloc.cc:136
136 /home/buildozer/aports/main/gcc/src/gcc-10.2.1_pre1/libstdc++-v3/libsupc++/eh_alloc.cc:  no such file or respo.

[anakrish]

@DylanWangWQF

Finally, make is successful!!

Great! That's good progress.

In my experience, now there will be a phase of finding out what functionality is not yet supported by OE and figuring out patches for it.

Program received signal SIGSEGV, Segmentation fault. std::__cxx11::basic_string<char, std::char_traits, std::allocator >::length (this=0x0) at /home/buildozer/aports/main/gcc/src/build/x86_64-alpine-linux-musl/libstdc++-v3/include/bits/basic_string.h:907

When there is a SEGV, the bracktrace can provide clues as to why the crash happens.

You mentioned the issue fix OE's locale implementation before. I don't know if this is the reason, but when I add this part in ecalls.cpp, got multiple definition of setlocale:

This is strange. I don't get this error in my tests: https://github.com/openenclave/openenclave/blob/3d6674a451608ea369f617fdd60f93cb5acdced2/tests/tools/apkman/cpp/boost/enc/enc.cpp#L20-L29

You can put a breakpoint in setlocale to see if it is getting hit, to determine if it is causing the crash by returning NULL.

In case you are not already doing so, I'd recommend that you build OE also from source. That way you will have maximum debugging information during development.

[DylanWangWQF]

Hi, @anakrish

You can put a breakpoint in setlocale to see if it is getting hit, to determine if it is causing the crash by returning NULL.

When I add this part in ecalls.cpp, make failed. multiple definition with oelibc (CmkaeLists.txt is shown below)

ld: /opt/openenclave/lib/openenclave/enclave/liboelibc.a(locale.c.o): in function `setlocale':
/home/dylan/openenclave/libc/locale.c:58: multiple definition of `setlocale'; CMakeFiles/enclave.dir/ecalls.cpp.o:ecalls.cpp:(.text.setlocale+0x0): first defined here

In case you are not already doing so, I'd recommend that you build OE also from source. That way you will have maximum debugging information during development.

I actually build OE from the source openenclave/openenclave on Ubuntu 18.04 with SGX1+FLC mode.

When there is a SEGV, the bracktrace can provide clues as to why the crash happens.

Now, when jump to this line in the debug mode, it directly jump to next line. Enclave Cpp: dipatcher.cpp

helib::Context e_context = helib::Context::readFrom(ess);

Debug Result

Get the string: |HE[
]HE||CN[
2+��������    @$@

       
�
��
��
�
(<
��

�
�
'
]CN|

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff23276fe in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::length() const ()
    at /home/buildozer/aports/main/gcc/src/gcc-10.2.1_pre1/libstdc++-v3/libsupc++/eh_alloc.cc:136
136 /home/buildozer/aports/main/gcc/src/gcc-10.2.1_pre1/libstdc++-v3/libsupc++/eh_alloc.cc: no such file or respo.
(gdb) 
_host_signal_handler (sig_num=0, sig_info=0x9, sig_data=0x0) at /home/dylan/openenclave/host/sgx/linux/exception.c:34
34  {
(gdb) 
35      ucontext_t* context = (ucontext_t*)sig_data;
(gdb) 
36      oe_host_exception_context_t host_context = {0};

The function helib::Context::readFrom(ess) invoked inside the enclave, it will return a Context object to the enclave. So I add some printf in HElib, Context.cpp for the debugging purpose. And it shows that until the return, the code runs correctly. HElib, Context.cpp, readFrom()

Context Context::readFrom(std::istream& str)
{
  printf("context readFrom from 0\n");
  Context::SerializableContent test_context_params = readParamsFrom(str);
  printf("context readFrom from 13\n");
  return Context(test_context_params);
}

Make Run Result

Get the string: |HE[
]HE||CN[
2+��������    @$@

       
�
��
��
�
(<
��

�
�
'
]CN|
Enclave: /home/dylan/mysamples/heenclave/enclave/dispatcher.cpp(45): Enclave: print the context: 
context readFrom from 0
...etc...
context readparamsfrom 12 0x7fba15a78bb0
context readFrom from 13
CMakeFiles/run.dir/build.make:57: recipe for target 'CMakeFiles/run' failed
make[3]: *** [CMakeFiles/run] Segmentation fault (core dumped)

Overall, helib::Context::readFrom(ess), this part runs correctly inside the enclave; So if this is due to the problem about assigning a objective inside the enclave? helib::Context e_context =

[DylanWangWQF]

In case you want to check the CMakeLists.txt:

target_include_directories(
  enclave
  PRIVATE ${CMAKE_CURRENT_BINARY_DIR}
          # C++
          ${APKMAN_ROOT}/usr/include/c++/10.2.1
          ${APKMAN_ROOT}/usr/include/c++/10.2.1/x86_64-alpine-linux-musl
          # C
          ${APKMAN_ROOT}/usr/include)

target_compile_options(
  enclave
  PRIVATE
  -Wno-shorten-64-to-32
  -Wno-bitwise-op-parentheses
  -Wno-shift-op-parentheses
  -Wno-sign-conversion
  -Wno-unused-parameter
  -Wno-implicit-int-conversion)

target_link_libraries(
    enclave 
    -nostdlib
    # Add library path and libraries.
    # -L$~/.apkman/alpine-fs/usr/lib
    -L${APKMAN_ROOT}/usr/lib
    libhelib.a
    libntl.a
    libgmp.a
    #libc.a
    openenclave::oecore
    libstdc++.a
    # for __umodti3
    -L${APKMAN_ROOT}/usr/lib/gcc/x86_64-alpine-linux-musl/10.2.1
    libgcc.a
    openenclave::oeenclave
    openenclave::oecrypto${OE_CRYPTO_LIB}
    openenclave::oelibcxx
    openenclave::oelibc
    # Use apkman's ld to avoid bug with Ubuntu 18.04's binutils.
    -fuse-ld=${CMAKE_CURRENT_SOURCE_DIR}/ld
    # Emit warnings for unresolved symbols.
    -Wl,--warn-unresolved-symbols)

[anakrish]

When I add this part in ecalls.cpp, make failed. multiple definition with oelibc (CmkaeLists.txt is shown below)

ld: /opt/openenclave/lib/openenclave/enclave/liboelibc.a(locale.c.o): in function `setlocale':
/home/dylan/openenclave/libc/locale.c:58: multiple definition of `setlocale'; CMakeFiles/enclave.dir/ecalls.cpp.o:ecalls.cpp:(.text.setlocale+0x0): first defined here

@DylanWangWQF I thought more about this. libc/locale.c has many functions defined. It is likely that in addition to setlocale, some other function from this file is referenced by the libs (HElib,ntl etc). This results in multiple definition error.

Since you are building OE from source, you can patch the definition of setlocale directly in libc/locale.c.

It is quite likelythat the SEGV issue you are running into is related to locales. One way to narrow down the issue would be to put breakpoints in all the functions in libc/locale.c and see if anything is not as expected by the libraries you are using.

So if this is due to the problem about assigning a objective inside the enclave? helib::Context e_context =

Quite likely, the constructor/copy-constructor of helib::Context is doing something that doesn't work out of the box within enclaves. Likely related to locales.

[DylanWangWQF]

Hi @anakrish

One way to narrow down the issue would be to put breakpoints in all the functions in libc/locale.c and see if anything is not as expected by the libraries you are using.

How to put breakpoints in libc/locale.c? After run /opt/openenclave/bin/oegdb -arg ./host/heenclave_host ./enclave/enclave.signed, cannot find this file even pending on future shared library load.

---

Since you are building OE from source, you can patch the definition of setlocale directly in libc/locale.c.

I directly change the source code in openenclave: openenclave/libc/locale.c (cannot modify as extern "C" char* setlocale(......)

static char _locale[256] = "C";
char* setlocale(int category, const char* locale)
{
    OE_UNUSED(category);
    if (locale == NULL)
        return _locale;
    sprintf(_locale, "%s", locale);
    return _locale;
}

openenclave/3rdparty/musl/muslinclude/locale.h

extern char *setlocale (int, const char *);

Then I build OE, copy /opt/openenclave to apkman path. But in doing so, still get previous segmentation fault

---

Since we get the error in the debug mode: More detailed debug information: debuginfo.log

Program received signal SIGSEGV, Segmentation fault.
0x00007fffe0327c8e in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::length() const ()
    at /home/buildozer/aports/main/gcc/src/gcc-10.2.1_pre1/libstdc++-v3/libsupc++/eh_alloc.cc:136
136 /home/buildozer/aports/main/gcc/src/gcc-10.2.1_pre1/libstdc++-v3/libsupc++/eh_alloc.cc: no such file or respo

If this is due to the compiler? undefined reference to std::__cxx11::basic_string Converting std::__cxx11::string to std::string

[anakrish]

Breakpoints via line numbers or function names ought to work. E.g: b locale.c:25 or b setlocale

If they don't work something is off.

[DylanWangWQF]

@anakrish I'm debugging the code. I will update the debug information later.

I have a question: Inside the enclave, can I construct the class constructor object (Context context)? Or should I pay attention to the allocated memory? Like the sample code, openenclave/samples/file-encryptor/enclave/mbedtls_src/encryptor.cpp

// initialize aes context
    m_aescontext = (struct aes_context*)malloc(sizeof(struct aes_context));
    if (m_aescontext == nullptr)
    {
        ret = 1;
        TRACE_ENCLAVE("allocate m_aescontext failed with %d", ret);
        goto exit;
    }

[anakrish]

@DylanWangWQF There is nothing unique about C++ constructors within the enclave. They work as they do normally outside the enclave. If the object being constructed requires a lot of dynamic memory, then you'd have the increase the number of heap pages in the enclave configuration.

[DylanWangWQF]

Hi, @anakrish . As I continue to debug in NTL library (the relevant function is shown in the next comment), I hit the issue Program received signal SIGTRAP, Trace/breakpoint trap., what can I learn from this clue? It's likely that something is wrong with thread. (Already try to compile libs with CXXFLAGS="-ftls-model=local-exec" or disable the thread in NTL and HElib) Here is the complete debug around SIGTRAP: debug.log Could you give me some suggestions?

NTL::details_pthread::push_node (p=0x7fffe0833460) at /home/dylan/Downloads/libwithclang/ntl_install/include/NTL/../NTL/tools.h:718
718 }
(gdb) 
NTL::UniqueID[abi:cxx11]() () at fileio.cpp:118
118    NTL_TLS_LOCAL_INIT(unsigned long, local_time, (time(0)));
(gdb) 
operator new (sz=24) at /home/buildozer/aports/main/gcc/src/gcc-10.2.1_pre1/libstdc++-v3/libsupc++/new_opnt.cc:42
42  /home/buildozer/aports/main/gcc/src/gcc-10.2.1_pre1/libstdc++-v3/libsupc++/new_opnt.cc: 没有那个文件或目录.
(gdb) 
operator new (sz=24) at /home/buildozer/aports/main/gcc/src/gcc-10.2.1_pre1/libstdc++-v3/libsupc++/new_op.cc:47
47  /home/buildozer/aports/main/gcc/src/gcc-10.2.1_pre1/libstdc++-v3/libsupc++/new_op.cc: 没有那个文件或目录.
(gdb) 
50  in /home/buildozer/aports/main/gcc/src/gcc-10.2.1_pre1/libstdc++-v3/libsupc++/new_op.cc
(gdb) 
malloc (size=24) at /home/dylan/openenclave/include/openenclave/corelibc/bits/malloc.h:10
10      return oe_malloc(size);
(gdb) 
oe_malloc (size=24) at /home/dylan/openenclave/enclave/core/malloc.c:23
23      void* p = oe_allocator_malloc(size);
(gdb) 
oe_allocator_malloc (size=24) at /home/dylan/openenclave/3rdparty/dlmalloc/allocator.c:89
89      return dlmalloc(size);
(gdb) 
dlmalloc (bytes=24) at /home/dylan/openenclave/3rdparty/dlmalloc/dlmalloc/malloc.c:4564
4564      ensure_initialization(); /* initialize in sys_alloc if not using locks */
(gdb) 
*************************************************OMIT***************************************
(gdb) 
oe_malloc (size=24) at /home/dylan/openenclave/enclave/core/malloc.c:25
25      if (!p && size)
(gdb) 
31      return p;
(gdb) 

time (t=0x0) at /home/dylan/openenclave/3rdparty/musl/musl/src/time/time.c:7
7       __clock_gettime(CLOCK_REALTIME, &ts);
(gdb) 
__clock_gettime (clk=0, ts=0x7fffefa7d818) at /home/dylan/openenclave/3rdparty/musl/musl/src/time/clock_gettime.c:94
94      r = __syscall(SYS_clock_gettime, clk, ts);
(gdb) 
oe_SYS_clock_gettime_impl (arg1=0, arg2=140737214142488) at /home/dylan/openenclave/libc/syscalls.c:47
47      clockid_t clock_id = (clockid_t)arg1;
(gdb) 
48      struct timespec* tp = (struct timespec*)arg2;
(gdb) 
49      int ret = -1;
(gdb) 
52      if (!tp)
(gdb) 
55      if (clock_id != CLOCK_REALTIME)
(gdb) 
62      if ((msec = oe_get_time()) == (uint64_t)-1)
(gdb) 
oe_get_time () at /home/dylan/openenclave/enclave/core/time.c:10
10      uint64_t ret = (uint64_t)-1;
(gdb) 
12      if (oe_ocall(OE_OCALL_GET_TIME, 0, &ret) != OE_OK)
(gdb) 
oe_ocall (func=32773, arg_in=0, arg_out=0x7fffefa7d6d0) at /home/dylan/openenclave/enclave/core/sgx/calls.c:702
702     oe_result_t result = OE_UNEXPECTED;
(gdb) 
703     oe_sgx_td_t* td = oe_sgx_get_td();
(gdb) 
oe_sgx_get_td () at /home/dylan/openenclave/enclave/core/sgx/td.c:171
171     asm("mov %%fs:0, %0" : "=r"(td));
(gdb) 
173     return td;
(gdb) 
oe_ocall (func=32773, arg_in=0, arg_out=0x7fffefa7d6d0) at /home/dylan/openenclave/enclave/core/sgx/calls.c:704
704     oe_callsite_t* callsite = td->callsites;
(gdb) 
708     if (__oe_enclave_status != OE_OK)
(gdb) 
712     if (!callsite)
(gdb) 
716     if (!td_initialized(td))
(gdb) 
td_initialized (td=0x7fffefa85000) at /home/dylan/openenclave/enclave/core/sgx/td.c:194
194     if (td && td->magic == TD_MAGIC && td->base.self_addr == (uint64_t)td)
(gdb) 
195         return true;
(gdb) 
198 }
(gdb) 
oe_ocall (func=32773, arg_in=0, arg_out=0x7fffefa7d6d0) at /home/dylan/openenclave/enclave/core/sgx/calls.c:725
725                  : [mxcsr] "m"(callsite->mxcsr),
(gdb) 
726                    [fcw] "m"(callsite->fcw),
(gdb) 
727                    [rflags] "m"(callsite->rflags)
(gdb) 
720     asm volatile("stmxcsr %[mxcsr] \n\t" // Save MXCSR
(gdb) 
731     if (oe_setjmp(&callsite->jmpbuf) == 0)
(gdb) 
oe_setjmp () at /home/dylan/openenclave/enclave/core/sgx/setjmp.S:20
*************************************************OMIT***************************************
(gdb) 
oe_setjmp () at /home/dylan/openenclave/enclave/core/sgx/setjmp.S:31
31      ret
(gdb) 
oe_ocall (func=32773, arg_in=0, arg_out=0x7fffefa7d6d0) at /home/dylan/openenclave/enclave/core/sgx/calls.c:734
734         _handle_exit(OE_CODE_OCALL, func, arg_in);
(gdb) 
_handle_exit (code=OE_CODE_OCALL, func=32773, arg=0) at /home/dylan/openenclave/enclave/core/sgx/calls.c:375
375     oe_exit_enclave(oe_make_call_arg1(code, func, 0, OE_OK), arg);
(gdb) 
oe_make_call_arg1 (code=OE_CODE_OCALL, func=OE_OCALL_GET_TIME, flags=0, result=OE_OK) at /home/dylan/openenclave/include/openenclave/internal/calls.h:118
118     return ((uint64_t)code << 48) | ((uint64_t)func << 32) |
(gdb) 
119            ((uint64_t)flags << 16) | ((uint64_t)result);
(gdb) 
118     return ((uint64_t)code << 48) | ((uint64_t)func << 32) |
(gdb) 
119            ((uint64_t)flags << 16) | ((uint64_t)result);
(gdb) 
118     return ((uint64_t)code << 48) | ((uint64_t)func << 32) |
__morestack (tcs=<optimized out>, aep=4905876, arg1=985183893323776, arg2=0, arg3=0x7fffffffd130, arg4=0x7fffffffd128, enclave=0x25d3190) at /home/dylan/openenclave/host/sgx/enter.c:225
225         arg1 = rdi;
(gdb) 
226         arg2 = rsi;
(gdb) 
230         if (code == OE_CODE_OCALL)
(gdb) 
232             __oe_host_stack_bridge(
(gdb) 

Program received signal SIGTRAP, Trace/breakpoint trap.
0x00000000004adba1 in __oe_host_stack_bridge (arg1=985183893323776, arg2=0, arg1_out=0x7fffffffcda8, arg2_out=0x7fffffffcda0, tcs=0x7fffefa80000, enclave=0x25d3190, ecall_context=0x25d3190)
    at /home/dylan/openenclave/host/sgx/enter.c:94
94  {
(gdb) 
98      bool debug = enclave->debug;
(gdb) 
99      if (debug)
(gdb) 
105         backup = *current;
(gdb) 
108         current->return_address = ecall_context->debug_eexit_rip;
(gdb) 
109         current->previous_rbp = ecall_context->debug_eexit_rbp;
(gdb) 
112     int ret = __oe_dispatch_ocall(arg1, arg2, arg1_out, arg2_out, tcs, enclave);
(gdb) 

Program received signal SIGTRAP, Trace/breakpoint trap.
0x0000000000497cc1 in __oe_dispatch_ocall (arg1=140737488342256, arg2=4906617, arg1_out=0x7fffffffcde0, arg2_out=0x4adc0c <__oe_host_stack_bridge+108>, tcs_=0x7fffffffccf0, enclave=0xbd3d07c03fd5cb00)
    at /home/dylan/openenclave/host/sgx/calls.c:421
421 {
(gdb) 
422     const oe_code_t code = oe_get_code_from_call_arg1(arg1);
Program received signal SIGTRAP, Trace/breakpoint trap.
0x00007ffff7ffb951 in clock_gettime ()
(gdb) 
Single stepping until exit from function clock_gettime,
which has no line number information.

Program received signal SIGTRAP, Trace/breakpoint trap.
0x00007ffff7ffb6c5 in ?? ()
(gdb) 
Cannot find bounds of current function

[DylanWangWQF]

Relevant code in NTL: NTL_TLS_LOCAL_INIT

// UniqueID:
//
// builds a string of the form cnt-time-clock-pid-tid, where
//   - cnt is a global counter
//   - time is the value returned by time(0)
//   - clock is the value returned by clock()
//   - pid is the value returned by getpid() (or "0" if getpid()
//        is not available)
//   - tid is the value returned by this_thread::get_id()
//        (or "0" if not using threads)
// each thread should have its own unique ID, which is guaranteed
//    to be unique across all threads in a process, and which
//    is hopefully unique across the entire system (but this
//    harder to guarantee)

const std::string& UniqueID()
{
   static AtomicCounter cnt; // a GLOBAL counter

   NTL_TLS_LOCAL(std::string, ID);

   NTL_TLS_LOCAL_INIT(bool, initialized, (false));
   NTL_TLS_LOCAL_INIT(unsigned long, local_cnt, (cnt.inc()));
   NTL_TLS_LOCAL_INIT(unsigned long, local_time, (time(0)));
   NTL_TLS_LOCAL_INIT(unsigned long, local_clock, (clock()));

   if (!initialized) {
      std::stringstream ss;
      ss << local_cnt << "-" << local_time << "-" 
         << local_clock << "-" << GetPID()  << "-" << CurrentThreadID();  
      ID = ss.str();
      initialized = true;
   }

   return ID;
}

[DylanWangWQF]

@anakrish I noticed that you talked about mystikos in https://github.com/openenclave/openenclave/issues/4110 For my case, I want to send the HE ciphertext to the enclave for decryption and encryption and return it to the host, so I need to write the EDL file to define the passed buffers. Is mystikos suitable for my case? Since in mystikos or sgx-lkl, we just put all of the code into the enclave.

BTW, if link the NTL and HElib in mystikos or sgx-lkl, still get the incompatible error? I found HElib is available in Alpine Linux Docker Container: https://github.com/IBM/fhe-toolkit-linux So can we conclude that the main problem is the unsupported functions in OE when linking HElib?