dimakuv
commented
2 years ago Golang on Linux uses raw system calls (raw syscall instructions, see https://www.felixcloutier.com/x86/syscall.html).
Intel SGX hardware disallows raw syscall instructions. This means that every syscall instruction invoked inside the SGX enclave results in a SIGILL ("Illegal Instruction", see also https://www.felixcloutier.com/x86/ud) exception. Exceptions in Intel SGX hardware are very expensive.
Gramine implements "trap-and-emulate" approach to catch SIGILL exceptions due to syscall instructions in the SGX enclave -- Gramine catches SIGILL, understands that the illegal instruction is a syscall, emulates this instruction via Gramine internal syscall logic, and then returns from the SIGILL exception back to normal execution. So from functionality perspective, the Golang application doesn't notice anything and happily continues its execution. But from performance perspective, there was a whole SIGILL exception -> AEX -> EENTER -> trap-and-emulate in Gramine -> EEXIT -> ERESUME flow, which is very expensive.
Why does this SIGILL exception happen only with Golang applications, you may ask. That's because all other applications/frameworks/runtimes (that I'm aware of) use dynamic linking against the C Standard library (Glibc). And all system call invokations happen through Glibc, so all raw syscall instructions are concentrated in Glibc code. Gramine provides a patched Glibc where all these raw syscall instructions are replaced with classic function calls into Gramine (see for example https://github.com/gramineproject/gramine/blob/v1.0/subprojects/packagefiles/glibc-2.34/glibc-2.34.patch). So when running applications with Gramine, there will be no raw syscall instructions because the patched Glibc will directly call Gramine syscall emulations. Thus, no SIGILL exceptions happen, and SGX performance is good.
You may wonder why Golang is so special (I think this is the only language/runtime that chooses to implement raw system calls instead of going through Glibc/Musl). It is actually a very interesting topic. Here are the relevant links:
- https://utcc.utoronto.ca/~cks/space/blog/programming/Go116OpenBSDUsesLibc
- https://utcc.utoronto.ca/~cks/space/blog/programming/GoCLibraryAPIIssues
- http://garrett.damore.org/2015/09/on-go-portability-and-system-interfaces.html
- https://groups.google.com/g/golang-nuts/c/uX8eUeyuuAY
- https://github.com/golang/go/issues/36435
The irony here is that to date (October 2021), Golang runtimes do not use raw syscall instructions and instead invoke system calls through the C Standard library on: Windows, MacOS, Illumos, Solaris, OpenBSD. But not on Linux. So if we would run Gramine-SGX on Windows or Mac, we would not have the awful performance penalty that we have currently on Linux.
As far as I know, there is currently no (simple) way to force Golang applications to invoke system calls through Glibc on Linux (please note that CGO_ENABLED Golang config does link against Glibc, but it does not remove raw syscall instructions from the Golang application). Some complex ways are:
- Statically analyze and replace
syscallinstructions in Golang binaries. Dynamically analyze and replace(UPDATE: In-place patching ofsyscallinstructions in Golang binaries.syscallinstructions is impossible in general case, though one may try ad-hoc solutions.)- Find an (unofficial) toolchain that produces Golang binaries that invoke system calls through Glibc.
- Wait for X years until Golang team allows invoking system calls through Glibc on Linux.
The Gramine team could explore solution (2) and incorporate some small dynamic binary rewriter into Gramine-SGX. But no promises here. On your side, you can google for solutions (1) and (3), or wait for solution (4).
yanzhichao
mkow
Description of the problem
Steps to reproduce
It's an web app which developed by beego. beego is an opensource MVC web framework which use golang (https://github.com/beego/beego/)
The app have a restfulAPI :
http://10.10.11.8:30002/marketplace/dp/v1/welfare/allIt's work fine without grahene-sgx , But it's QPS is very low when run in graphene-sgx , as follow pic show: need about 2 min to finish the requst.There must be some problems，could you give me some suggestion ？
graphene version：75962a812d686e839
the logs which one requst is as follow：