Simulation doesn't work with hello-rust

[elichai]

Hi, If I try to compile helloworld in SW mode it works but if I try hello-rust I get this error:

[+] Home dir is /root
[+] Open token file success! 
[+] Token file valid!
[-] Init Enclave Failed SGX_ERROR_MODE_INCOMPATIBLE!

Steps to reproduce :

$ docker pull baiduxlab/sgx-rust:latest
$ git clone https://github.com/baidu/rust-sgx-sdk.git
$ docker run -v $(pwd)/rust-sgx-sdk:/root/sgx -ti baiduxlab/sgx-rust:latest
$ cd sgx/samplecode/hello-rust
$ export SGX_MODE=SW # editing the Makefile gives the same effect
$ make
$ cd bin && ./app

Thanks!

[davidp94]

I encountered the same issue. @dingelish can help on that. It seems to be an issue with an import or the build.rs missing something.

in app/build.rs you can try to:

-        "SW" => println!("cargo:rustc-link-lib=dylib=sgx_urts_sim"),
-        "HW" => println!("cargo:rustc-link-lib=dylib=sgx_urts"),
-        _    => println!("cargo:rustc-link-lib=dylib=sgx_urts"), // Treat undefined as HW
+        "SW" => {
+            println!("cargo:rustc-link-lib=dylib=sgx_urts_sim");
+            println!("cargo:rustc-link-lib=dylib=sgx_uae_service_sim");
+        },
+        _    => { // HW by default
+            println!("cargo:rustc-link-lib=dylib=sgx_urts");
+            println!("cargo:rustc-link-lib=dylib=sgx_uae_service");
+        },

Tweak a little for the untrusted app, enabling auto-recompile on untrusted app.

In the Makefile

diff --git a/samplecode/tls-file/tlsserver/Makefile b/samplecode/tls-file/tlsserver/Makefile
index 999e4b1..36446ce 100644
--- a/samplecode/tls-file/tlsserver/Makefile
+++ b/samplecode/tls-file/tlsserver/Makefile
@@ -77,6 +77,7 @@ Enclave_EDL_Files := enclave/Enclave_t.c enclave/Enclave_t.h app/Enclave_u.c app
 ######## APP Settings ########

 App_Rust_Flags := --release
+App_SRC_Files := $(shell find app/ -type f -name '*.rs') $(shell find app/ -type f -name 'Cargo.toml')
 App_Include_Paths := -I ./app -I./include -I$(SGX_SDK)/include
 App_C_Flags := $(SGX_COMMON_CFLAGS) -fPIC -Wno-attributes $(App_Include_Paths)

@@ -107,11 +108,11 @@ RustEnclave_C_Files := $(wildcard ./enclave/*.c)
 RustEnclave_C_Objects := $(RustEnclave_C_Files:.c=.o)
 RustEnclave_Include_Paths := -I$(SGX_SDK)/include -I$(SGX_SDK)/include/tlibc -I$(SGX_SDK)/include/stlport -I$(SGX_SDK)/include/epid -I ./enclave -I./include

-RustEnclave_Link_Libs := -L$(CUSTOM_LIBRARY_PATH) -lcompiler-rt-patch -l$(Urts_Library_Name) -lenclave
+RustEnclave_Link_Libs := -L$(CUSTOM_LIBRARY_PATH) -lcompiler-rt-patch -lenclave
 RustEnclave_Compile_Flags := $(SGX_COMMON_CFLAGS) -nostdinc -fvisibility=hidden -fpie -fstack-protector $(RustEnclave_Include_Paths)
 RustEnclave_Link_Flags := $(SGX_COMMON_CFLAGS) -Wl,--no-undefined -nostdlib -nodefaultlibs -nostartfiles -L$(SGX_LIBRARY_PATH) \
    -Wl,--whole-archive -l$(Trts_Library_Name) -Wl,--no-whole-archive \
-   -Wl,--start-group -lsgx_tstdc -lsgx_tstdcxx -lsgx_tcxx -l$(Crypto_Library_Name) -l$(ProtectedFs_Library_Name) -l$(UProtectedFs_Library_Name) $(RustEnclave_Link_Libs) -Wl,--end-group \
+   -Wl,--start-group -lsgx_tstdc -lsgx_tcxx -l$(Crypto_Library_Name) -l$(Service_Library_Name) -l$(ProtectedFs_Library_Name) $(RustEnclave_Link_Libs) -Wl,--end-group \
    -Wl,-Bstatic -Wl,-Bsymbolic -Wl,--no-undefined \
    -Wl,-pie,-eenclave_entry -Wl,--export-dynamic  \
    -Wl,--defsym,__ImageBase=0 \
@@ -141,7 +142,7 @@ $(App_Enclave_u_Object): app/Enclave_u.o
    $(AR) rcsD $@ $^
    cp $(App_Enclave_u_Object) ./lib

-$(App_Name): $(App_Enclave_u_Object)
+$(App_Name): $(App_Enclave_u_Object) $(App_SRC_Files)
    @cd app && SGX_SDK=$(SGX_SDK) cargo build $(App_Rust_Flags)
    @echo "Cargo  =>  $@"
    cp $(App_Rust_Path)/app ./bin

[dingelish]

That's pretty much weird. I cannot reproduce the bug on all of my testbeds. Here are my logs:

ding@xlab-node17:~/rust-sgx-sdk$ git status .
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean
ding@xlab-node17:~/rust-sgx-sdk$ docker pull baiduxlab/sgx-rust:latest
latest: Pulling from baiduxlab/sgx-rust
Digest: sha256:f04e1fe27dba601ebe2605e29d8cc39448af2c820441a6f5c2c158990128dcfd
Status: Image is up to date for baiduxlab/sgx-rust:latest
ding@xlab-node17:~/rust-sgx-sdk$ docker run --rm -v $(pwd):/root/sgx -ti baiduxlab/sgx-rust:latest
root@a83634373522:~# cd sgx/samplecode/hello-rust/
root@a83634373522:~/sgx/samplecode/hello-rust# export SGX_MODE=SW
root@a83634373522:~/sgx/samplecode/hello-rust# make
/opt/sgxsdk/bin/x64/sgx_edger8r --trusted enclave/Enclave.edl --search-path /opt/sgxsdk/include --search-path ../../edl --trusted-dir enclave
/opt/sgxsdk/bin/x64/sgx_edger8r --untrusted enclave/Enclave.edl --search-path /opt/sgxsdk/include --search-path ../../edl --untrusted-dir app
GEN  =>  enclave/Enclave_t.c enclave/Enclave_t.h app/Enclave_u.c app/Enclave_u.h
CC   <=  enclave/Enclave_t.c
ar rcsD app/libEnclave_u.a app/Enclave_u.o
cp app/libEnclave_u.a ./lib
    Updating registry `https://github.com/rust-lang/crates.io-index`
 Downloading dirs v1.0.2
   Compiling app v1.0.0 (file:///root/sgx/samplecode/hello-rust/app)
   Compiling libc v0.2.42
   Compiling sgx_types v1.0.1 (file:///root/sgx/sgx_types)
   Compiling dirs v1.0.2
   Compiling sgx_urts v1.0.1 (file:///root/sgx/sgx_urts)
    Finished release [optimized] target(s) in 5.66s
Cargo  =>  bin/app
cp ./app/target/release/app ./bin
make -C ./enclave/
make[1]: Entering directory '/root/sgx/samplecode/hello-rust/enclave'
cargo build --release
   Compiling sgx_unwind v0.0.1 (file:///root/sgx/sgx_unwind)
   Compiling sgx_build_helper v0.1.0 (file:///root/sgx/build_helper)
   Compiling sgx_types v1.0.1 (file:///root/sgx/sgx_types)
   Compiling sgx_trts v1.0.1 (file:///root/sgx/sgx_trts)
   Compiling sgx_tstd v1.0.1 (file:///root/sgx/sgx_tstd)
   Compiling sgx_alloc v1.0.1 (file:///root/sgx/sgx_alloc)
   Compiling sgx_tprotected_fs v1.0.1 (file:///root/sgx/sgx_tprotected_fs)
   Compiling Helloworldsampleenclave v1.0.0 (file:///root/sgx/samplecode/hello-rust/enclave)
    Finished release [optimized] target(s) in 7.28s
cp ./target/release/libhelloworldsampleenclave.a ../lib/libenclave.a
make[1]: Leaving directory '/root/sgx/samplecode/hello-rust/enclave'
make -C ../../compiler-rt/ 2> /dev/null
make[1]: Entering directory '/root/sgx/compiler-rt'
cc -c -Wjump-misses-init -Wstrict-prototypes -Wunsuffixed-float-constants -fstack-protector -O2   -UDEBUG -DNDEBUG -Wall -Wextra -Winit-self -Wpointer-arith -Wreturn-type -Waddress -Wsequence-point -Wformat-security -Wmissing-include-dirs -Wfloat-equal -Wundef -Wshadow -Wcast-align -Wconversion -Wredundant-decls -DITT_ARCH_IA64 -ffreestanding -nostdinc -fvisibility=hidden -fpie -O2  -I../common/inc/ muloti4.c -o muloti4.o
ar rcsD libcompiler-rt-patch.a muloti4.o
make[1]: Leaving directory '/root/sgx/compiler-rt'
CC   <=  enclave/Enclave_t.c
cp ../../compiler-rt/libcompiler-rt-patch.a ./lib
LINK =>  enclave/enclave.so
<!-- Please refer to User's Guide for the explanation of each field -->
<EnclaveConfiguration>
    <ProdID>0</ProdID>
    <ISVSVN>0</ISVSVN>
    <StackMaxSize>0x40000</StackMaxSize>
    <HeapMaxSize>0x100000</HeapMaxSize>
    <TCSNum>1</TCSNum>
    <TCSPolicy>1</TCSPolicy>
    <DisableDebug>0</DisableDebug>
    <MiscSelect>0</MiscSelect>
    <MiscMask>0xFFFFFFFF</MiscMask>
</EnclaveConfiguration>
tcs_num 1, tcs_max_num 1, tcs_min_pool 1
The required memory is 1675264B.
Succeed.
SIGN =>  bin/enclave.signed.so
root@a83634373522:~/sgx/samplecode/hello-rust# cd bin/
root@a83634373522:~/sgx/samplecode/hello-rust/bin# ./app
[+] Home dir is /root
[-] Open token file /root/enclave.token error! Will create one.
[+] Saved updated launch token!
[+] Init Enclave Successful 2345052143618!
This is a normal world string passed into Enclave!
This is a in-Enclave Rust string!
[+] say_something success...

readelf -d would give something useful as follows:

root@a83634373522:~/sgx/samplecode/hello-rust/bin# readelf -d app

Dynamic section at offset 0x6da68 contains 31 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libsgx_urts_sim.so]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [librt.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [ld-linux-x86-64.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x000000000000000c (INIT)               0x6510
 0x000000000000000d (FINI)               0x5367c
 0x0000000000000019 (INIT_ARRAY)         0x269d40
 0x000000000000001b (INIT_ARRAYSZ)       16 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x269d50
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x2d0
 0x0000000000000005 (STRTAB)             0xef8
 0x0000000000000006 (SYMTAB)             0x388
 0x000000000000000a (STRSZ)              1970 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000015 (DEBUG)              0x0
 0x0000000000000003 (PLTGOT)             0x26dc98
 0x0000000000000007 (RELA)               0x18f0
 0x0000000000000008 (RELASZ)             19488 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x0000000000000018 (BIND_NOW)
 0x000000006ffffffb (FLAGS_1)            Flags: NOW PIE
 0x000000006ffffffe (VERNEED)            0x17a0
 0x000000006fffffff (VERNEEDNUM)         6
 0x000000006ffffff0 (VERSYM)             0x16aa
 0x000000006ffffff9 (RELACOUNT)          709
 0x0000000000000000 (NULL)               0x0

@elichai could you please provide the result of readelf -d app on your hello-rust project? Thanks!

[dingelish]

Hi @elichai ,

I wonder if you had done make before set SGX_MODE=SW, and then SGX_MODE=SW make without make clean at first? make cannot detect the mode change and won't re-compile the untrusted app without make clean.

[davidp94]

It looks like I had the issue because the Makefile was not setting the SGX_MODE to SW as it was already set. So once I explicitly set export SGX_MODE=SW in the terminal, it worked.

The readelf -a was returning that it was not using urts in simulation mode.

I pushed a change in Readme.md

2018-07-27 22:20 GMT+02:00 Yu Ding notifications@github.com:

Hi @elichai https://github.com/elichai ,

I wonder if you had done make before set SGX_MODE=SW, and then SGX_MODE=SW make without make clean at first? make cannot detect the mode change and won't re-compile the untrusted app without make clean.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/baidu/rust-sgx-sdk/issues/26#issuecomment-408528514, or mute the thread https://github.com/notifications/unsubscribe-auth/AYtbNdVAUu6FnpS0eWcXKwxYK3tb2XVlks5uK3XxgaJpZM4VheST .

[dingelish]

@davidp94 yeah that's might be the problem. Makefile set SGX_MODE ?= HW which means it only initiates SGX_MODE when its empty.

[dingelish]

Hi @elichai I can confirm that the project code is working well in SW mode. Please make clean and then make to switch from HW to SW mode.

[elichai]

Ok, exporting the SGX_MODE var was crucial. even on a new vm, new docker and everything, editing the Makefile isn't enough. I'm not sure if the Makefile doesn't assign that var or if it's not being passed to the build.rs (I tried putting a panic!() in the HW option in build.rs and it panics unless I do the export)

I still have problems doing this to an already exists big project. but I'll try using readelf -d to debug it. (Maybe it's because in my project i'm using version 1.0.0 for both your cargo libs and docker? ) Thanks!

[elichai]

So it looks like even though I added println!("cargo:rustc-link-lib=dylib=sgx_uae_service_sim"); to the build.rs file it doesn't use the simulation library of sgx_uae_service, here is readelf -d:

Dynamic section at offset 0xb28f0 contains 32 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libsgx_uae_service.so]
 0x0000000000000001 (NEEDED)             Shared library: [libsgx_urts_sim.so]
 0x0000000000000001 (NEEDED)             Shared library: [libzmq.so.5]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [librt.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [ld-linux-x86-64.so.2]
 0x000000000000000c (INIT)               0x8788
 0x000000000000000d (FINI)               0x885e0
 0x0000000000000019 (INIT_ARRAY)         0x2adfd8
 0x000000000000001b (INIT_ARRAYSZ)       16 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x2adfe8
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x2d0
 0x0000000000000005 (STRTAB)             0x1330
 0x0000000000000006 (SYMTAB)             0x388
 0x000000000000000a (STRSZ)              2481 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000015 (DEBUG)              0x0
 0x0000000000000003 (PLTGOT)             0x2b2b30
 0x0000000000000007 (RELA)               0x1f90
 0x0000000000000008 (RELASZ)             26616 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x0000000000000018 (BIND_NOW)           
 0x000000006ffffffb (FLAGS_1)            Flags: NOW PIE
 0x000000006ffffffe (VERNEED)            0x1e30
 0x000000006fffffff (VERNEEDNUM)         6
 0x000000006ffffff0 (VERSYM)             0x1ce2
 0x000000006ffffff9 (RELACOUNT)          963
 0x0000000000000000 (NULL)               0x0

@davidp94 maybe you have experience with uae_service?

[elichai]

Ok. So it turns out I had #[link(name = "sgx_uae_service")] in my code so it compiled with this instead of the simulation one. Now everything works. Thanks!