Registration fails with vault 1.15 or 1.16

[arnecls]

We recently started updating our vault 1.12 (on debian 11) installation to a more recent version (on debian 12) and found this plugin to not be working anymore.

The error happens when running

vault secrets enable -path=github -plugin-name=vault-plugin-secrets-github plugin

Vault 1.15 is complaining about a mismatched struct type

* 1 error occurred:
        * failed to decode plugin entry: json: cannot unmarshal string into Go struct field PluginRunner.type of type consts.PluginType

Vault 1.16 is complaining about an unrecognized remote plugin message:

Error enabling: Error making API request.

URL: POST http://127.0.0.1:8200/v1/sys/mounts/github
Code: 400. Errors:

* invalid backend version: 2 errors occurred:
        * Unrecognized remote plugin message: 
Failed to read any lines from plugin's stdout
This usually means
  the plugin was not compiled for this architecture,
  the plugin is missing dynamic-link libraries necessary to run,
  the plugin is not executable by this process due to file permissions, or
  the plugin failed to negotiate the initial go-plugin protocol handshake

Additional notes about plugin:
  Path: /opt/vault/plugins/vault-plugin-secrets-github
  Mode: -rwxr-xr-x
  Owner: 1000 [vault] (current: 1000 [vault])
  Group: 1001 [vault] (current: 1001 [vault])
  ELF architecture: EM_X86_64 (current architecture: amd64)

        * Unrecognized remote plugin message: 
Failed to read any lines from plugin's stdout
This usually means
  the plugin was not compiled for this architecture,
  the plugin is missing dynamic-link libraries necessary to run,
  the plugin is not executable by this process due to file permissions, or
  the plugin failed to negotiate the initial go-plugin protocol handshake

Additional notes about plugin:
  Path: /opt/vault/plugins/vault-plugin-secrets-github
  Mode: -rwxr-xr-x
  Owner: 1000 [vault] (current: 1000 [vault])
  Group: 1001 [vault] (current: 1001 [vault])
  ELF architecture: EM_X86_64 (current architecture: amd64)

Going through the list of suggested issues

the plugin was not compiled for this architecture

EM_X86_64 can be considered the same as amd64, so no

the plugin is missing dynamic-link libraries necessary to run

ldd yields not a dynamic executable so I guess this is fine.

the plugin is not executable by this process due to file permissions, or

The plugin has the same permission like another plugin that is working fine. The error output also suggests that there is no issue here.

the plugin failed to negotiate the initial go-plugin protocol handshake

I cannot check this, but the error from vault 1.15 might suggest that this is the actual issue.

I did not check older versions of vault yet, but I see that this plugin is compiled against vault API 1.12.11 which is also the version that we previously used and was also working.

[arnecls]

I can reproduce the same error as seen on 1.15 on 1.13.11, too.

[arnecls]

Bumping the github.com/hashicorp/vault/api dependency to v1.14.0 seems to fix it. I did increase a couple of other versions, too.

Upgrading to github.com/hashicorp/vault/sdk to v0.12.0 lead to an error, that's why I kept it at v0.11.1.

I compiled the binary with GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -ldflags="-s -w".

The go.mod file now looks like this

require (
    github.com/bradleyfalzon/ghinstallation v1.1.1
    github.com/hashicorp/go-hclog v1.6.3
    github.com/hashicorp/vault/api v1.14.0
    github.com/hashicorp/vault/sdk v0.11.1
    github.com/prometheus/client_golang v1.19.1
    github.com/prometheus/common v0.53.0
    gotest.tools v2.2.0+incompatible
)

[icep87]

@arnecls I'm getting the same error when installing it on Vault version 1.16.2. What Vault version are you running?

[arnecls]

I've been using 2.0.0 with vault 1.12 - that works. I've tried using 2.0.0 with vault 1.13.11 - that already does not work. I've tried 2.1.0 with vault 1.13, 1.15 and 1.16 - they all don't work.

I've created a PR with a fixed version. I think the vault/api package has a breaking change between 1.12 and 1.13. It might also be a linker flag, but that would not explain why version 2.0.0 is working with 1.12 but not 1.13.

[sover02]

I'm getting some inconsistencies with where this is working and where it is not. On my local dockerized instance, I'm not having any trouble enabling the engine in Vault version 1.16.2 (aarch64).

However, in my cloud deployment on x86_64, I see the above-mentioned issue. The architecture also affect the outcome for this bug?

➜  vault-local-dev ✗ vault secrets enable -path=github -plugin-name=vault-plugin-secrets-github plugin
Success! Enabled the vault-plugin-secrets-github secrets engine

Locally deployed Vault Server (docker)

root@d3c24ef94650:/# arch
aarch64
root@d3c24ef94650:/# vault version
Vault v1.16.2 (c6e4c2d4dc3b0d57791881b087c026e2f75a87cb), built 2024-04-22T16:25:54Z
root@d3c24ef94650:/# vault status
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    1
Threshold       1
Version         1.16.2
Build Date      2024-04-22T16:25:54Z
Storage Type    inmem
Cluster Name    vault-cluster-85fd8c95
Cluster ID      0a7febeb-1581-d95d-6284-1742fd76f874
HA Enabled      false

Local vault client (from Homebrew)

➜  vault-local-dev ✗ arch
arm64
➜  vault-local-dev ✗ vault version
Vault v1.14.4 ('ccdd48d1f7b95fc99fd11d67fc1c687576b338de+CHANGES'), built 2023-09-22T21:29:05Z