Hard to install as system package

[jonhoo]

I'm working on building a package for opensgx for Arch Linux (i.e. a PKGBUILD), but am running into a bunch of problems with hard-coded relative paths. For example, sgx.h has a hard-coded path to ../../qemu/target-i386/sgx.. Naturally, I can patch past all of these, but it would be great if the code base was less tightly coupled so that, say, sgx.h could be placed in /usr/include.

[tsgates]

Sorry, we never try to use it as a system package. We are happy to get patches if you'd prefer.

[jonhoo]

I'm working on a couple now. The biggest helpful change would be if make could produce a single sgx.o that sgx applications can be linked against. Currently, it's a big hassle to compile .sgx files -- you basically have to compile it using the Makefile target in user/.

[jonhoo]

First draft of package here: https://aur.archlinux.org/packages/opensgx-git Note the patches:

• opensgx-bin-paths.patch: makes opensgx use the commands sgx, sgx-tool, etc. directly now that they're in /usr/bin/
• opensgx-compile.patch: makes opensgx compile using the included sgx-compile wrapper instead of the Makefile (which needs all the rest of the build system to be present).
• demo-include-sgx.patch: makes hello.c use include <sgx.h>
• sgx-h-paths.patch: makes all SGX files use system includes (include <sgx/*.h> since headers are now in /usr/include/sgx)
• sgx-qemu-path.patch: patches the true path of the compiled qemu into sgx and sgx-dbg
• test-sh.patch: Patches test.sh to use the system-wide sgx command. This should probably also remove make in favor of sgx-compile, but I'm having some issues with test/test_kern (namely that it doesn't exist).

Using this install, I can successfully compile and sign hello.c, but executing it causes problems (log below). Any pointers as to where I might go from here would be welcome.

$ ls
hello.c
$ opensgx -k
$ opensgx -c hello.c
$ ln -sfn ../conf/ .
$ opensgx -s hello.sgx --key sign.key
$ opensgx hello.sgx hello.conf
ecode_size: 8192 edata_size: 4096 entry_offset: 0
K> init_epc(55): g_epc: 0x40008000
Q> encls_qemu_init(4748): set EPC pages 0x40007fff-0x405e4000
exininfo_t size: 4
grpsgx_t size: 192
DEBUG npages is 512
encalve size is 200000
K> sys_create_enclave(500): enclave addr: 0x40008000 (size: 0x200000 w/ secs = 0x40008000)
K> sys_create_enclave(511): add tcs 0x642000 (@0x40009000)
K> add_page_to_epc(322): add/copy 0x642000 -> 0x40009000
Q> sgx_eadd(3522): current cssa is 0
K> sys_create_enclave(517): add tls (fs/gs) pages: 0x63c000 (2 pages)
K> add_page_to_epc(322): add/copy 0x63c000 -> 0x4000a000
K> add_page_to_epc(322): add/copy 0x63c000 -> 0x4000b000
K> sys_create_enclave(523): add target code/data: 0x63e000 (3 pages)
K> add_page_to_epc(322): add/copy 0x63e000 -> 0x4000c000
K> add_page_to_epc(322): add/copy 0x63f000 -> 0x4000d000
K> add_page_to_epc(322): add/copy 0x640000 -> 0x4000e000
K> sys_create_enclave(529): add ssa pages: 0x63c000 (2 pages)
K> add_page_to_epc(322): add/copy 0x63c000 -> 0x4000f000
K> add_page_to_epc(322): add/copy 0x63c000 -> 0x40010000
K> sys_create_enclave(536): add stack pages: 0x63c000 (250 pages)
K> add_page_to_epc(322): add/copy 0x63c000 -> 0x40011000
K> add_page_to_epc(322): add/copy 0x63c000 -> 0x40012000
...
K> add_page_to_epc(322): add/copy 0x63c000 -> 0x40109000
K> add_page_to_epc(322): add/copy 0x63c000 -> 0x4010a000
DEBUG eps stack end is set as 0x4010a000
K> sys_create_enclave(543): add heap pages: 0x63c000 (100 pages)
K> add_page_to_epc(322): add/copy 0x63c000 -> 0x4010b000
DEBUG epc heap beg is set as 0x4010b000
K> add_page_to_epc(322): add/copy 0x63c000 -> 0x4010c000
K> add_page_to_epc(322): add/copy 0x63c000 -> 0x4010d000
...
K> add_page_to_epc(322): add/copy 0x63c000 -> 0x4016d000
K> add_page_to_epc(322): add/copy 0x63c000 -> 0x4016e000
DEBUG epc heap end is set as 0x4016efff
Q> sgx_einit(3744): Expected enclave measurement:--------testtest
0365177A30A8A060B64BB743DDC1E58E0C0AD965EBA529B6739644C787634CF0
Q> sgx_einit(3870): Expected launch key:
556CAD84C6FD89E853484875AFCC480B
Q> sgx_eenter(1546): aep: 0x40192a, tcs: 0x40009000
Q> sgx_eenter(1547): index_tcs: 1, mode64: 1
Q> sgx_eenter(1730): ssa: 0x4000f000 (size:576) base: 0x40008000
Q> sgx_eenter(1824): range: 0x40008000-200000
Q> sgx_eenter(1854): entry ptr: 0x4000c000 (base: 0x40008000, offset: 4000)
55 48 89 E5 53 48 83 EC 08 B8 00 00 00 00 E8 30 17 43 C0 B8 
Q> sgx_eenter(1882): old ursp: 0x40007ff858  changed rsp 0x4010a000 at eenter
Q> sgx_eenter(1887): transfer to rip: 0x4000c000 (rsp: 0x4010a000, rbp: 0x4010a000)
Q> sgx_eenter(1920): saved EXIT_EIP at eenter: 401879
Q> sgx_eenter(1924): async rip: 0x40192a
CPU_SIGNAL_HANDLER 11
user-exec.c Called exception_action
Exception Action
Debug Raise Exception: RBP: 40109ff8   RSP: 40109fe0 EIP: 43d743
POE_PAGE
MAPERR
qemu: uncaught target signal 11 (Segmentation fault) - core dumped
/usr/bin/sgx: line 10:  7377 Segmentation fault      (core dumped) $QEMU "$@"

[johnmwshih]

I'll check these patches. Thanks for your work!

[jonhoo]

You should specifically look at the build script here: https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=opensgx-git

[jonhoo]

I've been digging some more, and it seems the problem is in how I compile the .sgx binary. If I compile with ./opensgx -c user/demo/hello.c, but do all the other steps using the system-install, everything works (i.e. I can correctly sign and execute the binary). This is quite strange given that I get no compilation or linking errors. Does the code rely on some particular layout of the compiled file?

My system install compiles .sgx files using

cc -g -Wall -pedantic -Wno-unused-function -std=gnu1x -fno-stack-protector -fvisibility=hidden -Wl,-T,/usr/lib/sgx/sgx.lds $(find /usr/lib/sgx/ -type f -iname '*.o') $@

Where the files in /usr/lib/sgx/ come from:

cd user
all=$(make -dn demo/hello.sgx | grep 'Considering target file' | awk '{print $4}' | sed -e "s/'//g" -e 's/\.$//' | grep '\.o' | grep -v demo)
for f in $all; do
  cp "$f" -t "/usr/lib/sgx/$(dirname "$f")"
done
cd ..
cp "user/sgx.lds" "/usr/lib/sgx"

Which I believe should link in all the same files as are linked in by make user/demo/hello.sgx?

The diff between a failed (mine.log) and a successful (their.log) execution is given below:

$ diff -u their.log mine.log | grep -v "add_page_to_epc"
--- their.log   2015-10-14 18:14:49.312785805 -0400
+++ mine.log    2015-10-14 18:15:28.046119160 -0400
@@ -1,42 +1,42 @@
 K> init_epc(55): g_epc: 0x40008000
 Q> encls_qemu_init(4766): set EPC pages 0x40007fff-0x405e4000
 K> sys_create_enclave(500): enclave addr: 0x40008000 (size: 0x200000 w/ secs = 0x40008000)
-K> sys_create_enclave(511): add tcs 0x658000 (@0x40009000)
+K> sys_create_enclave(511): add tcs 0x642000 (@0x40009000)
 Q> sgx_eadd(3542): current cssa is 0
 K> sys_create_enclave(517): add tls (fs/gs) pages: 0x63c000 (2 pages)
-K> sys_create_enclave(523): add target code/data: 0x63e000 (25 pages)
+K> sys_create_enclave(523): add target code/data: 0x63e000 (3 pages)
 K> sys_create_enclave(529): add ssa pages: 0x63c000 (2 pages)
+K> sys_create_enclave(536): add stack pages: 0x63c000 (250 pages)
-K> sys_create_enclave(536): add stack pages: 0x63c000 (250 pages)
@@ -265,6 +265,7 @@
+K> sys_create_enclave(543): add heap pages: 0x63c000 (100 pages)
@@ -287,7 +288,6 @@
-K> sys_create_enclave(543): add heap pages: 0x63c000 (100 pages)
@@ -366,68 +366,21 @@
 Q> sgx_einit(3764): Expected enclave measurement:--------testtest
-A714074D487F1223BE5930DC554172183E886BE363973002AD7FE41AB18AFC60
+0365177A30A8A060B64BB743DDC1E58E0C0AD965EBA529B6739644C787634CF0
 Q> sgx_einit(3890): Expected launch key:
 556CAD84C6FD89E853484875AFCC480B
 Q> sgx_eenter(1562): aep: 0x40192a, tcs: 0x40009000
 Q> sgx_eenter(1563): index_tcs: 1, mode64: 1
-Q> sgx_eenter(1747): ssa: 0x40025000 (size:576) base: 0x40008000
+Q> sgx_eenter(1747): ssa: 0x4000f000 (size:576) base: 0x40008000
 Q> sgx_eenter(1841): range: 0x40008000-200000
-Q> sgx_eenter(1871): entry ptr: 0x400201bf (base: 0x40008000, offset: 181bf)
-55 48 89 E5 53 48 83 EC 08 B8 00 00 00 00 E8 2E BE FE FF B8 
-Q> sgx_eenter(1899): old ursp: 0x40007ff878     changed rsp 0x40120000 at eenter
-Q> sgx_eenter(1904): transfer to rip: 0x400201bf (rsp: 0x40120000, rbp: 0x40120000)
+Q> sgx_eenter(1871): entry ptr: 0x4000c000 (base: 0x40008000, offset: 4000)
+55 48 89 E5 53 48 83 EC 08 B8 00 00 00 00 E8 30 17 43 C0 B8 
+Q> sgx_eenter(1899): old ursp: 0x40007ff878     changed rsp 0x4010a000 at eenter
+Q> sgx_eenter(1904): transfer to rip: 0x4000c000 (rsp: 0x4010a000, rbp: 0x4010a000)
 Q> sgx_eenter(1937): saved EXIT_EIP at eenter: 401879
 Q> sgx_eenter(1941): async rip: 0x40192a
-Q> helper_mem_execute(1097): Executing memory in EPC (enclave:1): 0x4000c000
-Q> helper_mem_execute(1097): Executing memory in EPC (enclave:1): 0x4002065e
-Q> helper_mem_execute(1097): Executing memory in EPC (enclave:1): 0x4001e495
-Q> helper_mem_execute(1097): Executing memory in EPC (enclave:1): 0x4001e4ca
-Q> helper_mem_execute(1097): Executing memory in EPC (enclave:1): 0x4001e65f
-Q> helper_mem_execute(1097): Executing memory in EPC (enclave:1): 0x40020677
-Q> helper_mem_execute(1097): Executing memory in EPC (enclave:1): 0x400206d3
-Q> helper_mem_execute(1097): Executing memory in EPC (enclave:1): 0x400206b0
-Q> sgx_eexit(1991): Current ESP: 4011ff78   EBP: 4011ffa8
-Q> sgx_eexit(1998): EEXIT will return to provided addr: 408b49
-Q> sgx_eexit(2024): SSA ossa: 1D000 nssa 2 cssa 0
-Q> saveState(475): State will be store in 0x40025f40
-Q> sgx_eexit(2036): Addr followed by eexit is 400206CB
-L> sgx_trampoline(224): Trampoline Entered
-++++++ Function code: PUTS
-Q> sgx_eresume(2647): Current ESP: 40007fe818   EBP: 40007fe820
-Q> sgx_eresume(2948): Restart from here: 400206cb
-Q> sgx_eresume(2990): EBP: 4011ffa8  ESP: 4011ff78
-Q> helper_mem_execute(1097): Executing memory in EPC (enclave:1): 0x4000c01d
-Q> sgx_eexit(1991): Current ESP: 4011ffb8   EBP: 4011ffd8
-Q> sgx_eexit(2000): EEXIT will return saved EXIT_EIP: 401879
-ecode_size: 90112 edata_size: 12288 entry_offset: 141bf
-DEBUG npages is 512
-encalve size is 200000
-DEBUG eps stack end is set as 0x40120000
-DEBUG epc heap beg is set as 0x40121000
-DEBUG epc heap end is set as 0x40184fff
-hello sgx!
-
+POE_PAGE
+MAPERR
+qemu: uncaught target signal 11 (Segmentation fault) - core dumped
+/usr/bin/sgx: line 10:   844 Segmentation fault      (core dumped) $QEMU "$@"

[jonhoo]

FWIW, my compiled .sgx is here: http://jon.thesquareplanet.com/share/opensgx/hello.sgx The corresponding sign.key is here: http://jon.thesquareplanet.com/share/opensgx/sign.key

[johnmwshih]

I'm trying to reproduce your case. One quick question based on the observation: -K> sys_create_enclave(523): add target code/data: 0x63e000 (25 pages) +K> sys_create_enclave(523): add target code/data: 0x63e000 (3 pages) Is this due to your modification?

[jonhoo]

Presumably, but I'm not sure what would be causing it? Are you running Arch Linux? If so, you should be able to reproduce by running yaourt -S opensgx-git, and then trying to follow the instructions for demo.c (you'll have to change #include <sgx-lib.h> to #include <sgx.h>, and user opensgx instead of ./opensgx).

[johnmwshih]

I'm using Debian.. anyway, I'll figure it out!

[jonhoo]

Thanks! In theory, just doing the same things as are done in the PKGBUILD should allow you to reconstruct exactly the setup I have. If you have any questions at all, please let me know.

[johnmwshih]

@jonhoo after some digging, I think I found the root clause. The result it's not working is because in our customized linker script (sgx.lds), we use relative path to include object files in .enc_test & .enc_data sections. That's also the reason in your log there is only 3 pages copied into epc.

[jonhoo]

Aaaah, yes, that makes sense. Fixing this is going to be tricky, as it seems like you would need a separate linker script for every application that you compile. Perhaps I'll change sgx-compile to generate a temporary .lds file for the application being compiled, using sgx.lds as a sort of template..

Looking at the sections for .enc_text and .enc_data, I see the following files being included:

test/*.o(.text)
demo/*.o(.text)
polarssl_sgx/*.o(.text)
lib/*.o(.text)
*(.enc_text)
*Lib.o(.text)

Most of these are already in /usr/lib/sgx after my install, but I'm not currently copying over anything from test/; are any of these necessary for a real install? Also, I see that you include *(.enc_text) (and similarly for .enc_data). This is very general. Which files in particular are you trying to include from, so I can make sure to only include those in the final install?

[johnmwshih]

test/.o and demo/.o is to make sure the binary itself is included, so not including test/* should only affect test cases. In order to make test cases work, you may need to include test/* as well. The intention is to make every code & data (we static link every objects that are going to be used inside the enclave) into the .enc_text and .enc_data such that we can copy entire memory chunk into epc, as to avoid relocation & using shared library located outside the enclave.

[jonhoo]

Yes, that's fair. My question was specifically about what files are needed for the *(.enc_text) rule, as I'll need to copy those into the resulting package.

[johnmwshih]

.enc_text is defined in sgx-main.c, which is the entry point of enclave.

[jonhoo]

Woooo! It works! I've changed my sgx-compile to create an appropriate .lds script. It's a bit hacky, but I can now run demo.c successfully! See https://aur.archlinux.org/cgit/aur.git/tree/sgx-compile?h=opensgx-git.

For what it's worth, I did not need to copy any additional files into /usr/lib/sgx. The rule /usr/lib/sgx/*(.enc_data) seems to be sufficient.

[johnmwshih]

cool! Then I'll work on merging your patch :) Thanks!

[jonhoo]

That's great! Some of it might be hard to incorporate, as a couple of files now depend on absolute paths instead. The changes that use binaries from $PATH instead of through relative paths should be incorporated though, as you could simply have your Makefile add the appropriate directories to $PATH before running.

I think, in general, providing a make install target would be very useful, as it could do many of these things that my package now has to do. The linker script seems trickier, as it will actually vary for each application (since the application itself needs to be included), but maybe there's something more clever you can do there?

[jonhoo]

The more I think about it, the less ideal the linker script patching seems to be. I basically have to pre-process the argument list to sgx-compile to compile any .c files into .os, so that they can be added to the list of things to be included by the linker script. I don't know Link Script at all, but is there a way of referring to the resulting binary after compilation? Or at least the .os? That way we could avoid having to pass them all in the linker script source.

[jonhoo]

In fact, an even better solution would be if I could statically compile a binary, and then pass it to some script that will then rearrange the segments. This would both make it much easier to compile OpenSGX applications, and in the future, to allow developers to write applications in other languages.

[jonhoo]

Ah, 95a6d31da8825c10ba7776d2126904b3ca36a365 helps significantly! I've updated the Arch package to match, and it's much more straightforward now. I still need the hack that compiles the .os before linking, which is unfortunate, but it's still a step in the right direction. The new linker script also breaks applications that have their object files in subdirectories (since they won't be matched by the *), but that may not be much of a concern for now?