Open meithecatte opened 2 years ago
Just my two cents, for the syscall interface, there is another possibility: to patch the ELF file after the Go toolchain produces it. And this could be more convenient if the user only has the binary. For example, a tool could scan the whole ELF and find the pattern "mov ...; syscall", and rewrite it as a long jump to a new piece of code, to call Gramine then jump back.
@lejunzhu: This method is super unreliable and hard to maintain, we don't plan to go in this direction.
- wait for EDMM/SGXv2, which will allow dynamically allocating more threads (requires support in hardware, the Linux kernel and Gramine)
Are there any updates on this "waiting for EDMM/SGXv2 solution"?
And AFAIK, SGX2 only allows you to manage memory dynamically. I am not sure it can allow dynamically allocating more threads.
- wait for EDMM/SGXv2, which will allow dynamically allocating more threads (requires support in hardware, the Linux kernel and Gramine)
Are there any updates on this "waiting for EDMM/SGXv2 solution"?
Yes, it's been supported since Gramine v1.6. Pls see the related manifest option, issue https://github.com/gramineproject/gramine/issues/1223 and PR https://github.com/gramineproject/gramine/pull/1451 for details.
And AFAIK, SGX2 only allows you to manage memory dynamically. I am not sure it can allow dynamically allocating more threads.
Dynamic threading is achieved based on SGX2 that allows TCS to be added at runtime (specifically, by changing regular enclave pages into TCS pages). Pls see above and Section 3.4 of this EDMM paper.
Go (also known as Golang) is one of the languages currently popular for writing applications in. Ideally it would be possible to run those within Gramine. However, some issues within the Go runtime make that unreliable and slow.
Syscall interface
Gramine needs to be in charge of handling all syscalls made by the application. The recommended way to achieve this is to use a specific
call
instruction instead ofsyscall
. We carry patches for glibc and musl to handle this for most applications. However, because of an interesting decision made by the Go team, Go doesn't use the libc to issue syscalls. This is not the first time this has caused issues.For Gramine, the impact is that the fallback that traps the
syscall
instruction will be used. This causes a significant performance degradation, as many more context switches are required for each syscall. This is signalled by the following message printed during startup:As Go binaries are fully statically linked, the only solution seems to be patching the Go toolchain to create binaries with the Gramine-specific syscall ABI. Apart from the overhead of maintaining the patches, this would be somewhat annoying for users, as for most other stacks, the build system doesn't need to know about Gramine, and the binaries produced can be run on Linux directly for testing.
Go runtime thread count
Due to SGX limitations, we need to specify the total number of threads each process in the enclave might use (this is the
sgx.thread_num
manifest key). However, the number of threads the Go runtime may use is unbounded. The closest we have to limiting this is theGOMAXPROCS
environment variable, however this doesn't count the threads that are blocked on a syscall, so it doesn't solve the problem.Possible solutions:
sgx.thread_num
to a somewhat high number, cross your fingers and hope for the best (this is what's happening in practice right now)