Unable to mount secret, no such file or directory.

caddoo commented 6 years ago

When attempting to use the vault-ethereum plugin in a docker environment using the official vault docker image I am unable to mount the new plugin and receive a no such file or directory error.

Detailed Description

Here is the vault service in my docker compose file:

vault:
    build:
      context: ./
      dockerfile: vault.dockerfile
    ports:
      - "9200:8200"
    volumes:
      - ../vault/config:/vault/config
      - ../vault/server/data:/vault/data
      - ../vault/logs:/vault/logs
    environment:
      - VAULT_ADDR=http://192.168.1.60:9200
      - VAULT_LOCAL_CONFIG={"api_addr":"http://127.0.0.1:8200","plugin_directory":"/vault/plugins","backend":{"consul":{"scheme":"http", "address":"192.168.1.60:9500","advertise_addr":"http://192.168.1.60", "path":"vault/"}},"listener":{"tcp":{"address":"0.0.0.0:8200","tls_disable":"true"}}}
    command: server

I extend the official docker image to attempt to install the vault ethereum plugin:

FROM vault:latest

# Make new directory for plugins
RUN mkdir /vault/plugins && \

# Download binary for Ethereum plugin
RUN wget https://github.com/immutability-io/vault-ethereum/releases/download/v0.0.3/vault-ethereum_0.0.3_linux_amd64.zip && \
    unzip ./vault-ethereum_0.0.3_linux_amd64.zip && \
    rm ./vault-ethereum_0.0.3_linux_amd64.zip && \
    mv ./vault-ethereum /vault/plugins/vault-ethereum && \
    chown -R vault:vault /vault/plugins

I then perform the following operations (this is a fresh install of vault with consul as the backend).

docker exec -it development_vault_1 vault init
docker exec -it development_vault_1 vault unseal
....
docker exec -it development_vault_1 vault login token=...

docker exec -it development_vault_1 sha256sum /vault/plugins/vault-ethereum
Output:
fc4734c431ae0333fad4aa480f539aa8b9b4ddcfec89907c3577a8c3de70399e  /vault/plugins/vault-ethereum

vault write sys/plugins/catalog/ethereum-plugin sha_256="fc4734c431ae0333fad4aa480f539aa8b9b4ddcfec89907c3577a8c3de70399e" command="vault-ethereum"
Output:
Success! Data written to: sys/plugins/catalog/ethereum-plugin

docker exec -it development_vault_1 vault secrets enable -path=ethereum -plugin-name=ethereum-plugin plugin

Error:
Error enabling: Error making API request.

URL: POST http://192.168.1.60:9200/v1/sys/mounts/ethereum
Code: 400. Errors:

* fork/exec /vault/plugins/vault-ethereum: no such file or directory

ls -lrt /vault/ output

drwxr-xr-x    2 root     root             0 Feb 17 12:19 logs
drwxr-xr-x    2 root     root             0 Feb 17 12:19 data
drwxr-xr-x    2 root     root             0 Feb 19 19:15 config
drwxr-xr-x    1 vault    vault         4096 Feb 20 09:05 plugins
drwxr-xr-x    2 vault    vault         4096 Feb 20 09:09 file

ls -lrt /vault/plugins output

-rwxr-xr-x    1 root     root      19384144 Feb 20 09:05 vault-ethereum

Your Environment

Host OS: Windows 10 Docker: 17.12.0-ce-win47 stable Vault Docker: 0.9.3 Vault Ethereum: v0.0.3

caddoo commented 6 years ago

After some help from https://github.com/sirlatrom on the docker community slack, he determined because the base vault docker image uses alpine which uses musl and the plugin needs glibc it failed.

I then updated my vault dockerfile to install the following: https://github.com/sgerrand/alpine-pkg-glibc

The error now is

Error enabling: Error making API request.

URL: POST http://192.168.1.60:9200/v1/sys/mounts/ethereum
Code: 400. Errors:

* plugin exited before we could connect

cypherhat commented 6 years ago

What does your vault config look like? Specifically, is api_addr set?

cypherhat commented 6 years ago

My vault config (on Darwin) looks like this. Note: I am using TLS to talk to vault:

"default_lease_ttl" = "24h"

"max_lease_ttl" = "24h"

"backend" "file" {
  "path" = "/Users/immutability/etc/vault.d/data"
}

"api_addr" = "https://localhost:8200"

"listener" "tcp" {
  "address" = "localhost:8200"

  "tls_cert_file" = "/Users/immutability/etc/vault.d/vault.crt"
  "tls_client_ca_file" = "/Users/immutability/etc/vault.d/root.crt"
  "tls_key_file" = "/Users/immutability/etc/vault.d/vault.key"
}

"plugin_directory" = "/Users/immutability/etc/vault.d/vault_plugins"

caddoo commented 6 years ago

{ "api_addr":"http://127.0.0.1:8200", "plugin_directory":"/vault/plugins", "backend":{ "consul":{ "scheme":"http", "address":"192.168.1.60:9500", "advertise_addr":"http://192.168.1.60", "path":"vault/" } }, "listener":{ "tcp":{ "address":"0.0.0.0:8200", "tls_disable":"true" } } }

That is exactly the config that is running. If I use postman to interact with the API publically. I get the same error message:

POST /sys/mounts/ethereum

{
    "type": "plugin",
  "config": {
    "force_no_cache": true
  },
  "plugin_name": "ethereum-plugin"
}

Response:

{
    "errors": [
        "plugin exited before we could connect"
    ]
}

caddoo commented 6 years ago

@cypherhat Is there anywhere I can see more verbose errors to try and debug?

caddoo commented 6 years ago

Here is the output of ldd /vault/plugins/vault-ethereum

 /lib64/ld-linux-x86-64.so.2 (0x7f0661440000)
 libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7f0661440000)
 libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f0661440000)
Error relocating /vault/plugins/vault-ethereum: __fprintf_chk: symbol not found

cypherhat commented 6 years ago

Ok, let me review. Probably won't get to it until later this evening.

cypherhat commented 6 years ago

I am in the last stages of testing a release for the new go-ethereum release (Iceberg). I will cut a Darwin and Ubuntu-tested Linux build of that; then, I will start testing this on the official vault docker image. I apologize for the delay - I use a Xenial-based Vagrant box for testing the plugin rather than a Docker environment.

caddoo commented 6 years ago

@cypherhat The official vault docker image is based off this OS: https://alpinelinux.org FYI

caddoo commented 6 years ago

I think I need to build from source on Alpine, can you provide instructions on how to do this with Make or go build. Unfamiliar with both of these and I thought go build needs a src directory

caddoo commented 6 years ago

Due to the difference in the libraries between glibc and musl I recompiled in the docker file and seem to have got it working.

For reference is my new vault docker file:

FROM vault:latest

# Install build tools
RUN apk add --update alpine-sdk

# Make new directory for plugins
RUN mkdir /vault/plugins
# Download and compile Ethereum plugin
RUN apk update \
    && apk add go git
RUN go get github.com/immutability-io/vault-ethereum \
    && go build github.com/immutability-io/vault-ethereum
RUN mv /root/go/bin/vault-ethereum /vault/plugins/vault-etheruem

cypherhat commented 6 years ago

I meant to return to this. Apologies.

Can you PR your Dockerfile? That way we can let others work from the official Docker image. Thanks!

caddoo commented 6 years ago

Where would you like the Dockerfile? Also I'm still not up and running yet. Currently have this error:

vault write ethereum/accounts/test chain_id=1977 rpc_url=http://192.168.1.60:8545
Error writing data to ethereum/accounts/test: Error making API request.

URL: PUT http://192.168.1.60:9200/v1/ethereum/accounts/test
Code: 500. Errors:

* 1 error occurred:

* unexpected EOF

I've confirmed that from the Vault container I can create ethereum accounts via the JSON RPC:

curl 192.168.1.60:8545 -X POST --data '{"jsonrpc":"2.0","method":"personal_newAccount","params":["********"],"id":1}'
 -H "Content-Type: application/json"
{"jsonrpc":"2.0","id":1,"result":"0x589f9ae1df19d051661ee27b3c937269b62ddd6a"}

Any idea how to debug?

cypherhat commented 6 years ago

You can put it in the root directory of the repo. As regards your error, the plugin doesn't talk to the Ethereum node to create an account, so it is something related to Vault-Plugin communication most likely. I can give it a try this weekend - I had thought you had success.

cypherhat commented 6 years ago

Closing due to lack of activity.

zambien commented 5 years ago

According to @jefferai here: https://github.com/hashicorp/vault/issues/3417

plugins should be built with CGO_ENABLED=0

0x234 commented 5 years ago

@zambien this makes the build fail with:

go build github.com/ethereum/go-ethereum/crypto/secp256k1: build constraints exclude all Go files in /go/pkg/mod/github.com/ethereum/go-ethereum@v1.8.27/crypto/secp256k1
# github.com/ethereum/go-ethereum/rpc
/go/pkg/mod/github.com/ethereum/go-ethereum@v1.8.27/rpc/endpoints.go:96:19: undefined: ipcListen
/go/pkg/mod/github.com/ethereum/go-ethereum@v1.8.27/rpc/ipc.go:50:10: undefined: newIPCConnection

zambien commented 5 years ago

@jbunce, yep. Sorry about that. I forgot I had posted this here. The right way to handle this is to use some other means of cross-compilation if that is what you are going for (for example, compiling darwin in linux).

You can use xgo for this:

https://github.com/techknowlogick/xgo

Although perhaps that isn't maintained anymore so use git@github.com:techknowlogick/xgo.git.

The following will work assuming the following:

must compile OUTSIDE of GOPATH
must clear vendor folder

For example:

# get/build xgo with vendoring support
go get github.com/techknowlogick/xgo
cd $GOPATH/src/github.com/techknowlogick/xgo
go build .

# build with xgo
cd /tmp
git clone git@github.com:immutability-io/vault-ethereum.git
cd vault-ethereum
rm -rf vendor
$GOPATH/src/github.com/techknowlogick/xgo/xgo -out vault-ethereum  --targets=darwin/amd64,linux/amd64,windows/amd64 -deps=https://gmplib.org/download/gmp/gmp-6.0.0a.tar.bz2 .

Perhaps a better approach would be to use goreleaser cross-compilation but I haven't looked into that yet.

https://medium.com/@robdefeo/cross-compile-with-cgo-and-goreleaser-6af884731222

zambien commented 5 years ago

In order to resolve this I'd like to recommend the following: