search

hashicorp / vault

A tool for secrets management, encryption as a service, and privileged access management
https://www.vaultproject.io/
Other
31.03k stars 4.2k forks source link

Cannot enable plugin #19856

Closed liorwavebl closed 1 year ago

liorwavebl commented 1 year ago

Describe the bug I managed to write a custom plugin for vault. When I'm trying to enable it I'm getting an error message:

2023-03-30T15:31:42.328Z [DEBUG] core: spawning a new plugin process: plugin_name=customplugin id=4Ka60gLoIk
2023-03-30T15:31:42.400Z [INFO]  secrets.customplugin.customplugin_99db1adf.customplugin: configuring client automatic mTLS
2023-03-30T15:31:42.426Z [DEBUG] secrets.customplugin.customplugin_99db1adf.customplugin: starting plugin: path=/home/centos/vault/plugins/customplugin args=["/home/centos/vault/plugins/customplugin"]
2023-03-30T15:31:42.427Z [DEBUG] secrets.customplugin.customplugin_99db1adf.customplugin: plugin started: path=/home/centos/vault/plugins/customplugin pid=25172
2023-03-30T15:31:42.427Z [DEBUG] secrets.customplugin.customplugin_99db1adf.customplugin: waiting for RPC address: path=/home/centos/vault/plugins/customplugin
2023-03-30T15:31:42.432Z [DEBUG] secrets.customplugin.customplugin_99db1adf.customplugin.customplugin: 2023/03/30 15:31:42 cannot allocate memory
2023-03-30T15:31:42.442Z [ERROR] secrets.system.system_c3126942: error occurred during enable mount: path=path/

To Reproduce Steps to reproduce the behavior:

  1. Run vault operator unseal ...
  2. Run vault secrets enable -path=path -description="description" -plugin-name="customplugin" plugin

Expected behavior The plugin enabled successfully

Environment:

Vault server configuration file(s):

storage "file" {
  path = "<home_path>/data"
}

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_disable = "true"
}

ui = true
plugin_directory = "<home_path>/plugins"
heatherezell commented 1 year ago

Hello, The error might indicate an environmental issue. How much memory is available on your host?

liorwavebl commented 1 year ago

The server hash 2 GB RAM. When I'm using dev mode I can enable the plugin.

heatherezell commented 1 year ago

How much memory is available after starting Vault? You may need to provision more memory for the node.

liorwavebl commented 1 year ago

The vault consume 13% of the memory (245MB/2GB).

EDIT: I increased the memory to 8 GB and the problem still exists.

heatherezell commented 1 year ago

Please see this support KB article and see if it helps. Thanks!

liorwavebl commented 1 year ago

Disabling mlock solved the problem.

solidgoldbomb commented 1 year ago

I hit the same issue when switching from dev mode (which disables mlock) to a production deployment. Disabling mlock isn't something I wanted to do in a production environment. It wasn't obvious for me how to fix this so I'm dropping some pointers here in case it helps you or someone else to solve this more easily.

For those who want to keep mlock enabled, the "cannot allocate memory" issue can can be solved by setting the appropriate capability (cap_ipc_lock) on the plugin binary like this:

sudo setcap cap_ipc_lock=+ep /path/to/your/plugin

The docs do mention this requirement but it didn't turn up for me when searching for this issue: https://developer.hashicorp.com/vault/docs/plugins/plugin-architecture#plugin-execution

The error messages for this scenario aren't very helpful but here are a few bread crumbs that might tie the error message back to mlock in case this helps someone out in the future:

FWIW, this is where mlock is called in the plugin (via the plugin sdk): https://github.com/hashicorp/vault/blob/main/sdk/plugin/serve.go#L133 The mlock call is conditionally invoked in the plugin based on an environment variable (VAULT_PLUGIN_MLOCK_ENABLED) which is passed through from the vault binary when it executes the plugin. I believe this comes directly from the disable_mlock config parameter given to vault.