Closed DragonDev1906 closed 9 months ago
dot-asm
commented
9 months ago Ouch! At first I failed to reproduce it, but then managed to figure it out. [Newer?] clang --target=x86_64-fortanix-unknown-sgx doesn't pre-define __ELF__. As a quick test, can you confirm that you can build as env CFLAGS=-D__ELF__ cargo build ...?
BTW, no-threads is enforced on SGX, you don't need to engage it in your Cargo.toml, it's done automatically on the blst side.
dot-asm
commented
9 months ago [Newer?]
clang --target=x86_64-fortanix-unknown-sgxdoesn't pre-define__ELF__
Hmmm, it appears to be vice versa. The latest clang version does pre-define __ELF__, but not say 14... [And that's presumably how it slipped through. I've pulled latest version to test when worked on it.]
dot-asm
commented
9 months ago I've also figured out how to limit visibility of [dev-dependencies] used in benchmarks, so that it's now possible to link the test suite for fortanix-sgx. I wonder if it's possible to execute it. Could you try following on your SGX system?
git clone https://github.com/dot-asm/blst -b sgx-fix
cd blst/bindings/rust
cargo test --target=x86_64-fortanix-unknown-sgx
DragonDev1906
commented
9 months ago As a quick test, can you confirm that you can build as env CFLAGS=-DELF cargo build ...?
Yes, with env CFLAGS=-D__ELF__ cargo build it builds (both the minified example and the real project), I have not tried running it, yet.
The latest clang version does pre-define ELF, but not say 14... [And that's presumably how it slipped through. I've pulled latest version to test when worked on it.]
If it helps: This is the clang version I have installed on my dev machine (not the one that I'm running SGX applications on) (I assume that's the one cargo uses, too).
clang version 16.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/binInterestingly, I couldn't find clang on the machine I'm testing on, so I'm not sure how cargo is compiling the project there. I did not need env CFLAGS=-D__ELF__ for running the tests on that machine (Ubuntu 20.04.1). There the mini project in my first post compiles with and without env CFLAGS=-D__ELF__.
I wonder if it's possible to execute it. Could you try following on your SGX system?
I had to add the following to ~/.cargo/config as described in https://edp.fortanix.com/docs/installation/guide/
[target.x86_64-fortanix-unknown-sgx]
runner = "ftxsgx-runner-cargo"Most tests run successfully, but not all:
running 27 tests
test bindgen_test_layout_blst_fp ... ok
test bindgen_test_layout_blst_fp12 ... ok
test bindgen_test_layout_blst_fp2 ... ok
test bindgen_test_layout_blst_fp6 ... ok
test bindgen_test_layout_blst_fr ... ok
test bindgen_test_layout_blst_p1 ... ok
test bindgen_test_layout_blst_p1_affine ... ok
test bindgen_test_layout_blst_p2 ... ok
test bindgen_test_layout_blst_p2_affine ... ok
test bindgen_test_layout_blst_pairing ... ok
test bindgen_test_layout_blst_scalar ... ok
test bindgen_test_layout_blst_uniq ... ok
test bindgen_test_normal_types ... ok
test fp12_test::miller_loop_n ... ok
test min_pk::tests::test_aggregate ... ok
test min_pk::tests::test_multiple_agg_sigs ... ok
test min_pk::tests::test_serialization ... ok
test min_pk::tests::test_sign_n_verify ... ok
test min_sig::tests::test_aggregate ... ok
test min_sig::tests::test_multiple_agg_sigs ... ok
test min_sig::tests::test_serialization ... ok
test min_sig::tests::test_sign_n_verify ... ok
test p1_multi_scalar::test_add ... Attaching debugger
Error while executing SGX enclave.
Enclave panicked: Enclave triggered exception: SgxEnclaveRun { function: EResume, exception_vector: 14, exception_error_code: 6, exception_addr: 0x7f107621e000 }
ERROR: while running "ftxsgx-runner" "/home/jens/test/blst/bindings/rust/target/x86_64-fortanix-unknown-sgx/debug/deps/blst-a057e48fb8f28d27.sgxs" got exit status: 255
error: test failed, to rerun pass `--lib`
Caused by:
process didn't exit successfully: `ftxsgx-runner-cargo /home/jens/test/blst/bindings/rust/target/x86_64-fortanix-unknown-sgx/debug/deps/blst-a057e48fb8f28d27` (exit status: 255)I'm currently trying to give you a bit more information why the test failed (e.g. stack traces), but that's unfortunately not that easy in SGX (so far I've not managed to get debugging or stack traces on panics in the enclave to work as they should).
dot-asm
commented
9 months ago I had to add the [
runner = "ftxsgx-runner-cargo"] to ~/.cargo/config
Of course :-)
running 27 tests
Cool!
test p1_multi_scalar::test_add ... Attaching debugger Error while executing SGX enclave.
Try env CFLAGS=-DSCRATCH_LIMIT=46080 cargo test ...
dot-asm
commented
9 months ago Try
env CFLAGS=-DSCRATCH_LIMIT=46080 cargo test ...
Question is how large is the [default] stack in SGX enclave. The suggested scratch limit is verified (and exercised in CI) to work with 56KB stack. If SGX has smaller stack, then the limit can be scaled down accordingly. [And conversely, if stack is larger, one can increase it.] However, the smaller size results in lower performance, so one can wonder if it would be more sensible to find a way to increase the stack instead...
dot-asm
commented
9 months ago [And conversely, if stack is larger, one can increase it.]
So try even env CFLAGS=-DSCRATCH_LIMIT=92160 cargo test ...
DragonDev1906
commented
9 months ago Ah, the stack size issue again, I wish the error message for running out of stack size would be more distinct (I've hit that issue a few times already).
Question is how large is the [default] stack in SGX enclave.
0x20000 bytes (128KiB)
Try env CFLAGS=-DSCRATCH_LIMIT=46080 cargo test ...
Works, 27/27 Tests passed.
so one can wonder if it would be more sensible to find a way to increase the stack instead...
Luckily that's easy to do. The main disadvantage with increasing stack/heap size is that the enclave binary gets bigger and the enclave takes longer to start/load, other than that there isn't really a big downside, except that the enclave has to offload pages more often from EPC (Enclave Page Cache) to RAM, which can affect performance (offloading to RAM is slower than in normal applications because the data has to be encrypted and requires integrity protection). So when optimizing for performance in SGX this offloading to RAM also needs to be considered (together with the entire memory behavior of the application in the enclave). Especially since the scratch space is likely used in a random access pattern.
Overall I don't see this as a big limitation at the moment (see below), but it is something I we have to consider when creating enclaves if we care about performance.
[package.metadata.fortanix-sgx]
stack-size=0x200000 # 2MiBSo try even
env CFLAGS=-DSCRATCH_LIMIT=92160 cargo test ...
This one works with a stack-size of 144 KiB or larger.
Without the SCRATCH_LIMIT change the tests require 178 KiB to run through.
There is a hardware limitation on the address space that can be used in an enclave. This is an absolute upper limit of anything memory related (heap + stack + code + unusable space):
The (usually) bigger limitation is the max Processor Reserved Memory size (hardware limitation), which limits the max EPC size (can be specified in the bios)
Most platforms have either 128 MB or 256 MB of PRM. The exception are 3rd Generation Intel® Xeon® Scalable processors, which each support 512GB of PRM size, adding up to 1TB on a two-socket platform. ^1
As you can see, it really depends on the hardware how big a problem a large stack (or in this case scratch space) is. The EPC size is not a hard limit, but exceeding it too much will cause more cache misses (which are more expensive than normal L1-3 cache misses). I don't see a bit issue with the scratch size or increasing the stack size, since the EPC isn't a hard limitation and the CPU I'm testing on does have a large cache size.
dot-asm
commented
9 months ago Works, 27/27 Tests passed.
Yeah!!! :-) :-) :-)
that's easy to do [adjust the stack size].
I'm settling for the 45K scratch cap. While it appears to be possible to increase it to keep it working with the default 128K stack, the difference between 90K scratch working with 144K stack suggests that the stack utilization pattern is different on SGX, and it might be more appropriate to be more conservative ;-)
Regarding EPC and Hardware limitations (if you're interested)
Always! Thanks!
dot-asm
commented
9 months ago Fix is committed, the tip of the repo should work without any additional environment variables set. Thanks again!
DragonDev1906
commented
9 months ago Thank you, too! :) For fast responses, curiosity, maintaining this project and especially for supporting a non-standard target in a language the project (blst) itself isn't even written in.
I unfortunately have another issue using
blstfor thex86_64-fortanix-unknown-sgxtarget: The following code does not compile for the commits 78fee18 to 1514337 (current master). The last commit that compiled was 69b00c4:Command used:
cargo build --target x86_64-fortanix-unknown-sgxRustc version:rustc 1.73.0 (cc66ad468 2023-10-03)Here is the linker error I got: