[Tracking] Support for Hashicorp Vault

svenkata9 commented 1 year ago

[There is no issue in Gramine at present; This issue is filed only for tracking purposes]

The issue is with building Hashicorp Vault source as a dynamically linked binary instead of a statically linked library which comes by default.

Vault source code present at https://github.com/hashicorp/vault was previously using gox for building the binary. That did not support dynamically linked binary. Over the second half of last year vault scripts were updated to use go directly, but by default the binary is built as statically linked. As we know, this binary has inline syscalls that degrades performance while running in Gramine.

When I tried to introduce --linkshared option there are three dependent ingredients that fail to link properly. There are issues discussed in GitHub and other forums, but I couldn't find any resolution for those.

I also tried to build a hello binary with --linkshared and that builds and works without issues.

root@gramine-sgx:/home/svenkat9/vault/vault1# make dev
==> Checking that build is using go version >= 1.20...
==> Using go version 1.20.1...
==> Removing old directory...
==> Building...
# github.com/cespare/xxhash/v2
asm: xxhash_amd64.s:120: when dynamic linking, R15 is clobbered by a global variable access and is used here: 00092 (/home/svenkat9/go/pkg/mod/github.com/cespare/xxhash/v2@v2.1.1/xxhash_amd64.s:120)  ADDQ    R15, AX
asm: assembly failed
# github.com/cloudflare/circl/math/fp25519
asm: fp_amd64.s:68: when dynamic linking, R15 is clobbered by a global variable access and is used here: 00215 (/home/svenkat9/go/pkg/mod/github.com/cloudflare/circl@v1.1.0/math/fp25519/fp_amd64.s:68)        MULXQ    24(SI), AX, R15
asm: assembly failed
# github.com/cloudflare/circl/math/fp448
asm: fp_amd64.s:68: when dynamic linking, R15 is clobbered by a global variable access and is used here: 00350 (/home/svenkat9/go/pkg/mod/github.com/cloudflare/circl@v1.1.0/math/fp448/fp_amd64.s:68)  MULXQ   48(SI), AX, R15
asm: assembly failed
Makefile:36: recipe for target 'dev' failed
make: *** [dev] Error 1

kailun-qin commented 1 year ago

There are issues discussed in GitHub and other forums, but I couldn't find any resolution for those.

Pls kindly see my comments below.

github.com/cespare/xxhash/v2

For xxhash, pls consider to update it to v2.1.2+(https://github.com/cespare/xxhash/releases/tag/v2.1.2) as a fix had been introduced since then (see https://github.com/cespare/xxhash/issues/54 for details).

github.com/cloudflare/circl/math/fp25519 github.com/cloudflare/circl/math/fp448

For circl, pls consider to try its latest main branch. A fix addressing the issue has been introduced recently (https://github.com/cloudflare/circl/pull/407, https://github.com/cloudflare/circl/issues/391).

svenkata9 commented 1 year ago

Thanks, Kailun. Updating with your suggestions made that build error go away, but landed in another error related to azure go sdk.

# github.com/hashicorp/vault
2023/03/07 13:53:12 duplicated definition of symbol github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[go.shape.struct { github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v4.VirtualMachineAssessPatchesResult }]).PollUntilDone.func1, from github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v4 and github.com/hashicorp/vault-plugin-auth-azure
Makefile:36: recipe for target 'dev' failed

dimakuv commented 1 year ago

I don't know how to fix the latest build error (maybe some googling will help?).

But it looks like slowly but steadily Golang itself and its popular libraries/modules are embracing the "dynamic linking" world. This is very good news for Gramine :)

Why?: The main reason is that Go compiler uses R15 to access to global variables, so it alters its value (do not restore it). [ taken from https://github.com/cloudflare/circl/pull/407 ]

Go is full of such ad-hoc tricks...

svenkata9 commented 1 year ago

Yeah. I tried updating some modules in azure-sdk-for-go to their latest, but that ends up in other issues.

kailun-qin commented 1 year ago

I tried updating some modules in azure-sdk-for-go to their latest, but that ends up in other issues.

The latest related modules do not resolve the issue. A more straightforward one is to disable azure plugin in building, however it's a built-in vault plugin which IDK how to make it...

svenkata9 commented 1 year ago

Yes. I tried that as well, but I could not find a way to disable it.

svenkata9 commented 1 year ago

The hashicorp dependencies that I encountered for building a dynamically linked binary are (possibly) resolved now, but there are errors internal to vault source that prevents the shared object build. I raised a GitHub issue with vault yesterday, and they closed saying it won't be supported.

Error building with -linkshared flag · Issue #22228 · hashicorp/vault (github.com)

gramineproject / gramine

[Tracking] Support for Hashicorp Vault #1216