Unrecognised plugin message when built with Docker

[chrisbecke]

Describe the Bug

I am trying to build this plugin for inclusion in a vault image. When I attempt to enable the secrets plugin it responds with the message:

# vault secrets enable -path=oauth2 vault-plugin-secrets-oauthapp-v3.0.0-beta.3-linux-amd64
Error enabling: Error making API request.

URL: POST http://localhost:8200/v1/sys/mounts/oauth2
Code: 400. Errors:

* Unrecognized remote plugin message: 

This usually means that the plugin is either invalid or simply
needs to be recompiled to support the latest protocol.

Expected Behavior

One of

• the plugin to register
• a section in the readme detailing how to correctly build the plugin binary
• a section in the readme detailing how to enable plugin logging / other diagnostics.

Steps to Reproduce

1. Create docker-compose.yml and vault/Dockerfile.
2. Run docker compose build to build the vault image.
3. Run docker compose up -d to run the vault image.
4. Run docker compose exec vault sh to access the shell in the running vault container.
5. Run vault operator init -n 1 -t 1 to initialize the vault.
6. Run vault operator unseal <unseal-ket> to unseal the vault.
7. Run vault login <token> to set the login token.
8. Run sha256sum /vault/plugins/vault-plugin-secrets-oauthapp-v3.0.0-beta.3-linux-amd64 to get the hash of the binary.
9. Run vault plugin register -sha256=<hash> secret vault-plugin-secrets-oauthapp-v3.0.0-beta.3-linux-amd64 to register the plugin.
10. Run vault secrets enable -path=oauth2 vault-plugin-secrets-oauthapp-v3.0.0-beta.3-linux-amd64

Files

vault/Dockerfile

FROM golang:1.17 AS build

# xz tools is needed for the puppetlabs build process
RUN apt-get update && apt-get install -y xz-utils
# get the plugin source
RUN git clone https://github.com/puppetlabs/vault-plugin-secrets-oauthapp.git /src
WORKDIR /src
# Use the proper build scripts
RUN make dist PLUGIN_DIST_TARGETS=dist-bin-linux-amd64
WORKDIR /src/bin
# extract the built artifact
RUN tar -xf /src/artifacts/vault-plugin-secrets-oauthapp-v3.0.0-beta.3-linux-amd64.tar.xz

FROM vault:latest AS final

COPY --from=build --chown=vault:vault /src/bin/ /vault/plugins/

docker-compose.yml

services:
  vault:
    image: vault:oauthapp
    build:
      context: vault
    command: server
    cap_add:
      - IPC_LOCK
    environment:
      VAULT_ADDR: http://localhost:8200
      VAULT_API_ADDR: vault:8200
      VAULT_LOCAL_CONFIG: '{"storage": {"file": {"path": "/vault/file"}},"listener":{"tcp":{"address":"0.0.0.0:8200","tls_disable":true}},"plugin_directory":"/vault/plugins"}'
    ports:
    - 8200:8200

[impl]

Hi,

Thanks for reporting this! I was able to reproduce with your configuration. For some reason Vault won't show the underlying problem without the log level turned up, so adding VAULT_LOG_LEVEL: debug to the environment variables was helpful here. Then docker-compose logs shows:

vault_1  | 2021-09-08T21:44:20.853Z [DEBUG] secrets.vault-plugin-secrets-oauthapp-v3.0.0-beta.3-linux-amd64.vault-plugin-secrets-oauthapp-v3.0.0-beta.3-linux-amd64_fbda3210.vault-plugin-secrets-oauthapp-v3.0.0-beta.3-linux-amd64.vault-plugin-secrets-oauthapp-v3.0.0-beta.3-linux-amd64: 2021-09-08T21:44:20.853Z [ERROR] plugin shutting down: error="cannot allocate memory": metadata=true

This is because the plugin isn't configured with the ability to use mlock(2). From the documentation, we need to enable it on the plugin file as well:

# setcap cap_ipc_lock=+ep /vault/plugins/vault-plugin-secrets-oauthapp-v3.0.0-beta.3-linux-amd64

This solves your initial problem. Trying again, we now get the following error:

vault_1  | 2021-09-08T21:53:57.984Z [ERROR] secrets.vault-plugin-secrets-oauthapp-v3.0.0-beta.3-linux-amd64.vault-plugin-secrets-oauthapp-v3.0.0-beta.3-linux-amd64_8bd160c7.vault-plugin-secrets-oauthapp-v3.0.0-beta.3-linux-amd64.vault-plugin-secrets-oauthapp-v3.0.0-beta.3-linux-amd64: plugin tls init: error="error during token unwrap request: Put "vault:///v1/sys/wrapping/unwrap": unsupported protocol scheme "vault"" timestamp=2021-09-08T21:53:57.984Z

In this case, you forgot the URL scheme in your VAULT_API_ADDR. It should be set to http://vault:8200. After updating it, we can successfully write and read to the plugin:

/ # vault write -force oauth2/config
Success! Data written to: oauth2/config
/ # vault read oauth2/config
Key                                           Value
---                                           -----
tune_provider_timeout_expiry_leeway_factor    1.5
tune_provider_timeout_seconds                 30
tune_reap_check_interval_seconds              300
tune_reap_dry_run                             false
tune_reap_non_refreshable_seconds             86400
tune_reap_revoked_seconds                     3600
tune_reap_transient_error_attempts            10
tune_reap_transient_error_seconds             86400
tune_refresh_check_interval_seconds           60
tune_refresh_expiry_delta_factor              1.2

It would probably be helpful to mention turning up the log level and checking the memory locking configuration as troubleshooting steps explicitly. I'll turn this into a docs issue. Let me know if you have any other questions!