Error during building the vault-ethereum plugin

[aolol]

Detailed Description: I pulled the latest vault-ethereum plugin and tried to build it without success. We are using and building it in a docker environment but Im also not able to build it on my host machine. Around October 2018 it just worked fine. In the docker environment it's uses image FROM golang:1.9

Error: Step 3/14 : COPY ./vault/ethereum /go/src/github.com/immutability-io/vault-ethereum ---> 2b45f96ebb53 Step 4/14 : WORKDIR /go/src/github.com/immutability-io/vault-ethereum Removing intermediate container b038a1e526ed ---> 3ad2a13dd564 Step 5/14 : RUN go get ./ ---> Running in 229f06c45279 package github.com/ethereum/go-ethereum/crypto/sha3: cannot find package "github.com/ethereum/go-ethereum/crypto/sha3" in any of: /usr/local/go/src/github.com/ethereum/go-ethereum/crypto/sha3 (from $GOROOT) /go/src/github.com/ethereum/go-ethereum/crypto/sha3 (from $GOPATH) ERROR: Service 'vault' failed to build: The command '/bin/sh -c go get ./' returned a non-zero code: 1

Environment: OS: Mac OS Mojave 10.14.2 (18C54) Docker version: Version 18.03.1-ce-mac65 (24312) GO version: go version go1.11.1 darwin/amd64 vault-ethereum: latest

[cypherhat]

Seems that go-ethereum made some changes. Let me realign:

 rename {crypto => vendor/golang.org/x/crypto}/sha3/doc.go (96%)
 rename {crypto => vendor/golang.org/x/crypto}/sha3/hashes.go (58%)
 create mode 100644 vendor/golang.org/x/crypto/sha3/hashes_generic.go
 rename {crypto => vendor/golang.org/x/crypto}/sha3/keccakf.go (100%)
 rename {crypto => vendor/golang.org/x/crypto}/sha3/keccakf_amd64.go (88%)
 rename {crypto => vendor/golang.org/x/crypto}/sha3/keccakf_amd64.s (100%)
 rename {crypto => vendor/golang.org/x/crypto}/sha3/register.go (100%)
 rename {crypto => vendor/golang.org/x/crypto}/sha3/sha3.go (100%)
 create mode 100644 vendor/golang.org/x/crypto/sha3/sha3_s390x.go
 create mode 100644 vendor/golang.org/x/crypto/sha3/sha3_s390x.s
 rename {crypto => vendor/golang.org/x/crypto}/sha3/shake.go (83%)
 create mode 100644 vendor/golang.org/x/crypto/sha3/shake_generic.go
 rename {crypto => vendor/golang.org/x/crypto}/sha3/xor.go (100%)
 rename {crypto => vendor/golang.org/x/crypto}/sha3/xor_generic.go (100%)
 rename {crypto => vendor/golang.org/x/crypto}/sha3/xor_unaligned.go (100%)

[cypherhat]

So, you should be good now. Upstream go-ethereum removed github.com/ethereum/go-ethereum/crypto/sha3 which contained NewKeccak256. This function is now in golang.org/x/crypto/sha3 as NewLegacyKeccak256.

Note: I haven't rebuilt and released the plugin with these changes yet. I am doing integration tests with Vault 1.0.2 at present. I will release once those are done.

[richard-mauri]

Maybe this project should use a dependency manager tool like https://github.com/golang/go/wiki/Modules ?

Would you accept a PR to factor that in? (I'm new to this project, so it may take a little acclimation first)

I hope I'm not getting too hard ahead of myself, but here is the output of a command the fetches the transitive closure of all dependencies using minor release versions. There is indeed some trouble here and we really should consider locking down to more specific/concrete versions. I'm updating this issue with rather verbose comments just to share the thought process and workflow...

Below is output dependding on latest latest path at this moment in time. I'll follow-up with another attempt using more stable dependency versions. I am using go1.11.4

Richards-MacBook:vault-ethereum richardmauri$ go get -u 
go: finding github.com/miguelmota/go-coinmarketcap/pro latest
go: finding github.com/hashicorp/vault/helper/cidrutil latest
go: finding github.com/hashicorp/vault/logical/framework latest
go: finding github.com/ethereum/go-ethereum/core/types latest
go: finding github.com/ethereum/go-ethereum/common latest
go: finding github.com/ethereum/go-ethereum/accounts/abi/bind latest
go: finding github.com/ethereum/go-ethereum/common/math latest
go: finding github.com/ethereum/go-ethereum/ethclient latest
go: finding github.com/ethereum/go-ethereum/common/hexutil latest
go: finding github.com/ethereum/go-ethereum v1.8.22
go: finding github.com/miguelmota/go-coinmarketcap latest
go: downloading github.com/miguelmota/go-coinmarketcap v0.0.0-20190203133305-bf4a9c70cc07
go: finding github.com/hashicorp/vault/helper latest
go: finding github.com/hashicorp/vault/logical latest
go: finding github.com/hashicorp/vault v1.0.2
go: finding golang.org/x/crypto/scrypt latest
go: downloading github.com/hashicorp/vault v1.0.2
go: downloading github.com/ethereum/go-ethereum v1.8.22
go: finding golang.org/x/crypto latest
go: downloading golang.org/x/crypto v0.0.0-20190131182504-b8fe1690c613
go: finding golang.org/x/crypto/sha3 latest
go: finding github.com/hashicorp/vault/helper/pluginutil latest
go: finding github.com/ethereum/go-ethereum/core latest
go: finding github.com/ethereum/go-ethereum/accounts/abi latest
go: finding github.com/ethereum/go-ethereum/accounts latest
go: finding github.com/hashicorp/vault/logical/plugin latest
go: finding github.com/ethereum/go-ethereum/accounts/keystore latest
go: finding github.com/ethereum/go-ethereum/crypto latest
go: finding github.com/shopspring/decimal latest
go: finding github.com/sethvargo/go-diceware/diceware latest
go: finding github.com/pborman/uuid v1.2.0
go: downloading github.com/shopspring/decimal v0.0.0-20180709203117-cd690d0c9e24
go: downloading github.com/pborman/uuid v1.2.0
go: finding github.com/sethvargo/go-diceware latest
go: downloading github.com/sethvargo/go-diceware v0.0.0-20181024230814-74428ac65346
go: finding github.com/google/uuid v1.0.0
go: finding golang.org/x/net v0.0.0-20180215212450-dc948dff8834
go: finding github.com/anaskhan96/soup v1.0.1
go: finding github.com/hashicorp/go-hclog latest
go: finding github.com/deckarep/golang-set v1.7.1
go: finding github.com/hashicorp/go-sockaddr v1.0.1
go: finding github.com/go-stack/stack v1.8.0
go: finding github.com/hashicorp/go-immutable-radix v1.0.0
go: finding github.com/hashicorp/go-version v1.1.0
go: finding github.com/mitchellh/go-testing-interface v1.0.0
go: finding github.com/hashicorp/go-uuid v1.0.1
go: finding google.golang.org/grpc v1.18.0
go: finding github.com/hashicorp/golang-lru v0.5.0
go: downloading github.com/hashicorp/go-hclog v0.0.0-20190109152822-4783caec6f2e
go: downloading github.com/deckarep/golang-set v1.7.1
go: downloading github.com/hashicorp/go-sockaddr v1.0.1
go: downloading github.com/go-stack/stack v1.8.0
go: downloading github.com/mitchellh/go-testing-interface v1.0.0
go: downloading github.com/hashicorp/go-version v1.1.0
go: downloading github.com/hashicorp/go-immutable-radix v1.0.0
go: downloading github.com/hashicorp/go-uuid v1.0.1
go: downloading github.com/hashicorp/golang-lru v0.5.0
go: finding github.com/aristanetworks/goarista/monotime latest
go: downloading golang.org/x/net v0.0.0-20180215212450-dc948dff8834
go: downloading github.com/google/uuid v1.0.0
go: downloading google.golang.org/grpc v1.18.0
go: finding github.com/golang/protobuf/proto latest
go: finding github.com/golang/protobuf/ptypes latest
go: finding github.com/hashicorp/go-plugin latest
go: finding github.com/rjeczalik/notify v0.9.2
go: finding github.com/SermoDigital/jose/jws latest
go: finding github.com/aristanetworks/goarista latest
go: downloading github.com/aristanetworks/goarista v0.0.0-20190201223726-e2363f69bde9
go: finding github.com/allegro/bigcache v1.1.0
go: downloading github.com/hashicorp/go-plugin v0.0.0-20190129155509-362c99b11937
go: finding github.com/golang/protobuf v1.2.0
go: downloading github.com/rjeczalik/notify v0.9.2
go: finding github.com/SermoDigital/jose v0.9.1
go: finding github.com/golang/snappy latest
go: finding github.com/golang/protobuf/ptypes/timestamp latest
go: downloading github.com/allegro/bigcache v1.1.0
go: finding github.com/armon/go-radix v1.0.0
go: downloading github.com/SermoDigital/jose v0.9.1
go: downloading github.com/golang/protobuf v1.2.0
go: finding github.com/syndtr/goleveldb/leveldb/filter latest
go: finding github.com/syndtr/goleveldb/leveldb/iterator latest
go: finding github.com/rs/cors v1.6.0
go: finding github.com/syndtr/goleveldb/leveldb latest
go: finding github.com/syndtr/goleveldb/leveldb/util latest
go: finding github.com/syndtr/goleveldb/leveldb/errors latest
go: downloading github.com/armon/go-radix v1.0.0
go: finding github.com/syndtr/goleveldb/leveldb/opt latest
go: downloading github.com/rs/cors v1.6.0
go: finding github.com/ryanuber/go-glob v1.0.0
go: finding github.com/syndtr/goleveldb latest
go: downloading github.com/syndtr/goleveldb v0.0.0-20190203031304-2f17a3356c66
go: finding github.com/hashicorp/go-rootcerts v1.0.0
go: finding github.com/hashicorp/go-retryablehttp v0.5.2
go: downloading github.com/ryanuber/go-glob v1.0.0
go: finding github.com/hashicorp/hcl/hcl/ast latest
go: downloading github.com/hashicorp/go-retryablehttp v0.5.2
go: finding golang.org/x/time/rate latest
go: finding github.com/hashicorp/hcl/hcl latest
go: downloading github.com/hashicorp/go-rootcerts v1.0.0
go: finding golang.org/x/time latest
go: finding golang.org/x/text/unicode/norm latest
go: finding golang.org/x/text/unicode/bidi latest
go: finding golang.org/x/text/secure/bidirule latest
go: finding golang.org/x/text/unicode latest
go: finding golang.org/x/text/secure latest
go: finding golang.org/x/sys v0.0.0-20180926160741-c2ed4eda69e7
go: finding github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b
go: finding golang.org/x/tools v0.0.0-20180828015842-6cd1fcedba52
go: finding github.com/client9/misspell v0.3.4
go: finding github.com/stretchr/testify v1.2.2
go: finding golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be
go: finding github.com/pmezard/go-difflib v1.0.0
go: finding golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4
go: finding github.com/kisielk/gotool v1.0.0
go: finding cloud.google.com/go v0.26.0
go: finding golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f
go: finding github.com/hashicorp/yamux v0.0.0-20180604194846-3520598351bb
go: finding google.golang.org/grpc v1.14.0
go: finding github.com/mitchellh/go-testing-interface v0.0.0-20171004221916-a61a99592b77
go: finding github.com/mitchellh/cli v1.0.0
go: finding honnef.co/go/tools v0.0.0-20180728063816-88497007e858
go: finding github.com/stretchr/testify v1.3.0
go: finding github.com/ryanuber/columnize v2.1.0+incompatible
go: finding golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3
go: finding github.com/hashicorp/go-uuid v1.0.0
go: finding github.com/golang/mock v1.1.1
go: finding github.com/mattn/go-colorable v0.0.9
go: finding github.com/fatih/color v1.7.0
go: finding github.com/oklog/run v1.0.0
go: finding github.com/mitchellh/go-wordwrap v1.0.0
go: finding golang.org/x/net v0.0.0-20180826012351-8a410e7b638d
go: finding golang.org/x/sys v0.0.0-20190129075346-302c3dd5f1cc
go: finding golang.org/x/sys v0.0.0-20180830151530-49385e6e1522
go: finding github.com/bgentry/speakeasy v0.1.0
go: finding google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8
go: finding golang.org/x/sys v0.0.0-20180823144017-11551d06cbcc
go: finding github.com/armon/go-radix v0.0.0-20180808171621-7fddfc383310
go: finding google.golang.org/appengine v1.1.0
go: finding github.com/davecgh/go-spew v1.1.0
go: finding github.com/stretchr/objx v0.1.0
go: finding github.com/hashicorp/go-hclog v0.0.0-20180709165350-ff2cf002a8dd
go: finding github.com/mattn/go-isatty v0.0.3
go: finding github.com/posener/complete v1.1.1
go: finding github.com/onsi/gomega v1.4.3
go: finding github.com/onsi/ginkgo v1.7.0
go: finding github.com/onsi/ginkgo v1.6.0
go: finding golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e
go: finding golang.org/x/net v0.0.0-20180906233101-161cd47e91fd
go: finding github.com/hpcloud/tail v1.0.0
go: finding github.com/fsnotify/fsnotify v1.4.7
go: finding gopkg.in/fsnotify.v1 v1.4.7
go: finding gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7
go: finding gopkg.in/yaml.v2 v2.2.1
go: finding gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405
go: downloading golang.org/x/net v0.0.0-20180906233101-161cd47e91fd
go: downloading github.com/hashicorp/yamux v0.0.0-20180604194846-3520598351bb
go: downloading github.com/oklog/run v1.0.0
go: downloading google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8
# github.com/immutability-io/vault-ethereum
./path_convert.go:206:25: client.CryptocurrencyListingsLatest undefined (type *coinmarketcap.Client has no field or method CryptocurrencyListingsLatest)
./path_convert.go:206:56: undefined: coinmarketcap.CryptocurrencyListingsLatestOptions

Let's try to blindly pull in patch versions (nope, we're going to to lock down more consciously):

Richards-MacBook:vault-ethereum richardmauri$ go get -u=patch
go: finding gopkg.in/yaml.v2 v2.2.2
go: finding github.com/stretchr/objx v0.1.1
go: finding github.com/posener/complete v1.1.2
go: finding github.com/mattn/go-isatty v0.0.4
# github.com/immutability-io/vault-ethereum
./path_convert.go:206:25: client.CryptocurrencyListingsLatest undefined (type *coinmarketcap.Client has no field or method CryptocurrencyListingsLatest)
./path_convert.go:206:56: undefined: coinmarketcap.CryptocurrencyListingsLatestOptions

Ok, so github.com/miguelmota/go-coinmarketcap is troublesome for us. I see there are no releases and there are no tags even! (boy this is a wild west ecosystem) This project will be in build hell until we lock down versions to what actually works.

FYI: I opened an issue with the root cause breakage.

The following commit reference to the broken dependency results in a sane build of this project.

Richards-MacBook:vault-ethereum richardmauri$ go get github.com/miguelmota/go-coinmarketcap/pro/v1@b0b8fc089275fa7220beb289fd44448dd3b0c982
go: finding github.com/miguelmota/go-coinmarketcap/pro b0b8fc089275fa7220beb289fd44448dd3b0c982
go: finding github.com/miguelmota/go-coinmarketcap b0b8fc089275fa7220beb289fd44448dd3b0c982
go: downloading github.com/miguelmota/go-coinmarketcap v0.0.0-20190131021944-b0b8fc089275
Richards-MacBook:vault-ethereum richardmauri$ go install
go: finding github.com/miguelmota/go-coinmarketcap v0.0.0-20190131021944-b0b8fc089275```

I tried to push a branch with go.mod,go.sum,README.md to fix the build using a generalized approach, but I lack permission. I am not savy enough to even QA this project, so care must be exercised by PR reviewers to ensure the builds really works.

Richards-MacBook:vault-ethereum richardmauri$  git push --set-upstream origin use-go-mod-and-fix-build
remote: Permission to immutability-io/vault-ethereum.git denied to richard-mauri.
fatal: unable to access 'https://github.com/immutability-io/vault-ethereum.git/': The requested URL returned error: 403

[cypherhat]

@richard-mauri I would gladly accept a PR for better dependency management. I was gonna use govendor but Vault and Ethereum move so fast. I haven't had a chance to get acclimated to go modules yet, but if you can help, it would be greatly appreciated.

[miguelmota]

suggest using go mod

export GO111MODULE=on
go mod init
go mod vendor

will submit pr shortly

[miguelmota]

submitted pr https://github.com/immutability-io/vault-ethereum/pull/57 but then realized there's an existing pr https://github.com/immutability-io/vault-ethereum/pull/55

[richard-mauri]

I tried to use the new go-coinmarketcap v0.1.0 release tag and the ame compile error happens. See https://github.com/miguelmota/go-coinmarketcap/issues/4#issuecomment-460649625

[miguelmota]

@richard-mauri there's a slight fix that needs to be made in this package. i've included it in my pr https://github.com/immutability-io/vault-ethereum/pull/57/files#diff-f49de6eb10bbf5e37308f19d72032d22L206

[richard-mauri]

55 includes change to use the new API so once that PR is merged this ticket can be closed

[cypherhat]

Fixed. https://github.com/immutability-io/vault-ethereum/releases/tag/v0.2.8