Hashicorp Vault plugin does not execute when Vault deployed via container

[ilsaloving]

I am trying to deploy a Hashicorp Vault container that uses Keeper as the backend.

When I try to register the plugin, I get this:

/vault/plugins # vault plugin register -sha256=84003377be76db1768067962bc44c6c913328e39eb663f8cfc68254a8aab056a secret vault-plugin-secrets-ksm
Success! Registered plugin: vault-plugin-secrets-ksm

But the vault server log shows this:

vault-server_1  | 2024-04-19T15:08:05.333Z [DEBUG] core: attempting to load backend plugin: name=vault-plugin-secrets-ksm
vault-server_1  | 2024-04-19T15:08:05.333Z [DEBUG] core: spawning a new plugin process: plugin_name=vault-plugin-secrets-ksm id=MTtEKVpZ6o
vault-server_1  | 2024-04-19T15:08:05.454Z [DEBUG] core: failed to dispense v5 backend plugin: name=vault-plugin-secrets-ksm error="fork/exec /vault/plugins/vault-plugin-secrets-ksm: no such file or directory"
vault-server_1  | 2024-04-19T15:08:05.550Z [DEBUG] core: failed to dispense v4 backend plugin: name=vault-plugin-secrets-ksm error="fork/exec /vault/plugins/vault-plugin-secrets-ksm: no such file or directory"
vault-server_1  | 2024-04-19T15:08:05.551Z [WARN]  core: Error determining plugin version:
vault-server_1  |   error=
vault-server_1  |   | 1 error occurred:
vault-server_1  |   | \t* fork/exec /vault/plugins/vault-plugin-secrets-ksm: no such file or directory
vault-server_1  |   |
vault-server_1  |

Sure enough, when I try to execute it myself, I get this: /vault/plugins # ./vault-plugin-secrets-ksm /bin/sh: ./vault-plugin-secrets-ksm: not found

If I deploy vault on a regular VM, I do not have this issue. After a great deal of research (These were particularly useful: https://github.com/hashicorp/vault/issues/8009, https://github.com/hashicorp/vault/issues/17250, https://megamorf.gitlab.io/2019/09/08/alpine-go-builds-with-cgo-enabled/), the issue apparently has something to do with alpine linux used in the container, and that it handles dynamic libraries differently than expected.

The solution is to compile the plugin statically linked instead of dynamically.

[ilsaloving]

After much trial and error, I gave up trying to get it to work /w Hashicorp's image. I rolled my own container of Vault using Ubuntu as the base instead of alpine.

Apart from the below warning, everything works, but Keeper devs have some updating to do. For anyone interested, I attached my Dockerfile to this ticket: https://github.com/hashicorp/vault/issues/26550

vault-server_1  | 2024-04-19T21:48:25.565Z [DEBUG] core: attempting to load backend plugin: name=vault-plugin-secrets-ksm
vault-server_1  | 2024-04-19T21:48:25.565Z [DEBUG] core: spawning a new plugin process: plugin_name=vault-plugin-secrets-ksm id=2HcWuiKynM
vault-server_1  | 2024-04-19T21:48:25.929Z [DEBUG] core: failed to dispense v5 backend plugin: name=vault-plugin-secrets-ksm
vault-server_1  |   error=
vault-server_1  |   | Unrecognized remote plugin message:
vault-server_1  |   | Failed to read any lines from plugin's stdout
vault-server_1  |   | This usually means
vault-server_1  |   |   the plugin was not compiled for this architecture,
vault-server_1  |   |   the plugin is missing dynamic-link libraries necessary to run,
vault-server_1  |   |   the plugin is not executable by this process due to file permissions, or
vault-server_1  |   |   the plugin failed to negotiate the initial go-plugin protocol handshake
vault-server_1  |   |
vault-server_1  |   | Additional notes about plugin:
vault-server_1  |   |   Path: /opt/vault/plugins/vault-plugin-secrets-ksm
vault-server_1  |   |   Mode: -rwxr-xr-x
vault-server_1  |   |   Owner: 0 [root] (current: 0 [root])
vault-server_1  |   |   Group: 0 [root] (current: 0 [root])
vault-server_1  |   |   ELF architecture: EM_X86_64 (current architecture: amd64)
vault-server_1  |
vault-server_1  | 2024-04-19T21:48:26.038Z [DEBUG] core: successfully dispensed v4 backend plugin: name=vault-plugin-secrets-ksm