search

immutability-io / vault-ethereum

A plugin that turns Vault into an Ethereum wallet.
244 stars 66 forks source link

plugin fails if disable_mlock is enabled #4

Closed shayangz closed 6 years ago

shayangz commented 6 years ago

Detailed Description

In production, it's recommended to run Vault with disable_mlock set to false. However, in our testing it appears that the plugin fails to work with this setting in the recommended production state.

Here is the error code:

Mar 15 16:14:08 vagrant-vbox-1 vault[20896]: 2018/03/15 16:14:08.989697 [DEBUG] plugin.metadata: starting plugin: path=/etc/vault/plugins/vault-ethereum args=[/etc/vault/plugins/vault-ethereum]
Mar 15 16:14:08 vagrant-vbox-1 vault[20896]: 2018/03/15 16:14:08.990385 [DEBUG] plugin.metadata: waiting for RPC address: path=/etc/vault/plugins/vault-ethereum
Mar 15 16:14:09 vagrant-vbox-1 vault[20896]: 2018/03/15 16:14:09.013390 [DEBUG] plugin.metadata.vault-ethereum: 2018/03/15 16:14:09 cannot allocate memory
Mar 15 16:14:09 vagrant-vbox-1 vault[20896]: 2018/03/15 16:14:09.014564 [DEBUG] plugin.metadata: plugin process exited: path=/etc/vault/plugins/vault-ethereum
Mar 15 16:14:09 vagrant-vbox-1 vault[20896]: 2018/03/15 16:14:09.014791 [ERROR] sys: mount failed: path=ethereum/ error=plugin exited before we could connect

This is with vault 0.9.5 and the latest vault-ethereum off of master.

Here is vault config file

{
  "plugin_directory": "/etc/vault/plugins",
  "api_addr": "http://127.0.0.1:8200",
  "disable_mlock": false,
  "listener": {
    "tcp": {
      "address": "127.0.0.1:8200",
      "tls_disable": "true"
    }
  },
  "storage": {
    "consul": {
      "token": "some-token-xxx",
      "address": "127.0.0.1:8500"
    }
  }
}

Exactly the same configuration with just disable_mlock set to true works fine.

Please note that vault in dev mode starts with disable_mlock set to true which is why we didn't see this problem in our dev environment testing.

Any thoughts?

cypherhat commented 6 years ago

Can you give me your Vagrantfile?

cypherhat commented 6 years ago

Also, what is your host OS?

cypherhat commented 6 years ago

Just saw this - https://www.vaultproject.io/docs/configuration/index.html#disable_mlock

sudo setcap cap_ipc_lock=+ep $(readlink -f $(which vault))

Testing with that...

shayangz commented 6 years ago

my Vagrant file is tied to a bunch internal infrastructure.

I am testing on Ubuntu 16.04

$ uname -a
Linux vagrant-vbox-1 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
cypherhat commented 6 years ago

Ok. After I saw the setcap call, I figured out what was going on...

Basically, this sudo setcap cap_ipc_lock=+ep $(readlink -f $(which vault)) allows the vault process to read memory without being root. So, the only thing that needs to be done is to allow the plugin to do the same. (Since it is a separate process.)

$ sudo setcap cap_ipc_lock=+ep $(readlink -f /home/vagrant/etc/vault.d/vault_plugins/vault-ethereum)

Of course, replace /home/vagrant/etc/vault.d/vault_plugins/vault-ethereum with the location of your plugin.

Please let me know if that works for you.

shayangz commented 6 years ago

ah clever! That got me over the cannot allocate memory error. But now I am hitting this error when starting vault:

Mar 15 23:29:53 vagrant-vbox-1 vault[8870]: 2018/03/15 23:29:53.252994 [DEBUG] plugin.metadata: starting plugin: path=/etc/vault/plugins/vault-ethereum args=[/etc/vault/plugins/vault-ethereum]
Mar 15 23:29:53 vagrant-vbox-1 vault[8870]: 2018/03/15 23:29:53.253577 [DEBUG] plugin.metadata: waiting for RPC address: path=/etc/vault/plugins/vault-ethereum
Mar 15 23:29:53 vagrant-vbox-1 vault[8870]: 2018/03/15 23:29:53.283952 [DEBUG] plugin.metadata.vault-ethereum: plugin address: address=/tmp/plugin983040137 network=unix
Mar 15 23:29:53 vagrant-vbox-1 vault[8870]: 2018/03/15 23:29:53.285918 [WARN ] plugin.metadata: error closing client during Kill: err=rpc error: code = Canceled desc = grpc: the client connection is closing
Mar 15 23:29:53 vagrant-vbox-1 vault[8870]: 2018/03/15 23:29:53.289647 [DEBUG] plugin.metadata: plugin process exited: path=/etc/vault/plugins/vault-ethereum

and attempting to write to accounts gets me this error

$ vault write ethereum/test4 chain_id=1977
Error writing data to ethereum/accounts/test4: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/ethereum/accounts/test4
Code: 500. Errors:

* 1 error occurred:

* rpc error: code = Unavailable desc = transport is closing

corresponding to this in the vault logs:

Mar 15 23:31:16 vagrant-vbox-1 vault[8870]: 2018/03/15 23:31:16.621148 [DEBUG] plugin: plugin process exited: path=/etc/vault/plugins/vault-ethereum
Mar 15 23:31:53vagrant-vbox-1 vault[8870]: 2018/03/15 23:31:53.300739 [WARN ] plugin: error closing client during Kill: err=rpc error: code = Canceled desc = grpc: the client connection is closing
Mar 15 23:31:53 vagrant-vbox-1 vault[8870]: 2018/03/15 23:31:53.470534 [DEBUG] plugin: starting plugin: path=/etc/vault/plugins/vault-ethereum args=[/etc/vault/plugins/vault-ethereum]
Mar 15 23:31:53 vagrant-vbox-1 vault[8870]: 2018/03/15 23:31:53.472178 [DEBUG] plugin: waiting for RPC address: path=/etc/vault/plugins/vault-ethereum
Mar 15 23:31:53 vagrant-vbox-1 vault[8870]: 2018/03/15 23:31:53.558112 [DEBUG] plugin.vault-ethereum: plugin address: address=/tmp/plugin093062371 network=unix

Incidentally, running vault read ethereum/test1 on an existing account works fine.

again setting disable_mlock to true makes the error go away.

cypherhat commented 6 years ago

I was able to successfully execute vault write ethereum/test4 chain_id=1977 with mlock with the above setcap commands. However, I am running with TLS.

If you want to run in a production setting, then you should use TLS. I haven't tested much without TLS of late because I never use vault without it.

You can see how I configure vault with TLS (and install the plugin) here

shayangz commented 6 years ago

fair enough on TLS. Once I turned TLS on, everything works fine.

I see the follow error logs when starting vault, but plugin seems to still function.

Mar 16 00:36:36 vault-i-x vault[28310]: 2018/03/16 00:36:36.248995 [DEBUG] plugin.metadata: starting plugin: path=/etc/vault/plugins/vault-ethereum args=[/etc/vault/plugins/vault-ethereum --ca-cert=/etc/ssl/certs/ca.pem --client-cert=/etc/vault/ssl/certs/vault.crt --client-key=/etc/vault/ssl/private/vault.key]
Mar 16 00:36:36 vault-i-x vault[28310]: 2018/03/16 00:36:36.249913 [DEBUG] plugin.metadata: waiting for RPC address: path=/etc/vault/plugins/vault-ethereum
Mar 16 00:36:36 vault-i-x vault[28310]: 2018/03/16 00:36:36.289086 [DEBUG] plugin.metadata.vault-ethereum: plugin address: address=/tmp/plugin033840308 network=unix
Mar 16 00:36:36 vault-i-x vault[28310]: 2018/03/16 00:36:36.296467 [WARN ] plugin.metadata: error closing client during Kill: err=rpc error: code = Canceled desc = grpc: the client connection is closing
Mar 16 00:36:36 vault-i-x vault[28310]: 2018/03/16 00:36:36.301168 [DEBUG] plugin.metadata: plugin process exited: path=/etc/vault/plugins/vault-ethereum