search

hyperledger-labs / minifabric

Do fabric network the right and easy way.
Apache License 2.0
302 stars 166 forks source link

Latest minifab on MacOS does not bring networks up: fails to install 'simple' chaincode #67

Closed edkaz closed 3 years ago

edkaz commented 3 years ago

Hello The latest minifab does gives an error and stops at the chaincode installation step. The problem seems to be that the chaincode installer is expecting there to be dev containers but for there are none. I apologise for the copious output but have included it for completeness.

Thanks - Ed.

Environment MacOS 10.15.6 Centos 7.7

Steps to Recreate the Problem

  1. minifab cleanup -o org1.example.com This command succeeds and the ./vars directory is empty.
  2. minifab up -o org1.example.com -s couchdb -c testchannel -e true Output: 
    
Minifab Execution Context:
FABRIC_RELEASE=2.2.0
CHANNEL_NAME=testchannel
PEER_DATABASE_TYPE=couchdb
CHAINCODE_LANGUAGE=go
CHAINCODE_NAME=simple
CHAINCODE_VERSION=1.0
CHAINCODE_INIT_REQUIRED=true
CHAINCODE_PARAMETERS="init","a","200","b","300"
CHAINCODE_PRIVATE=false
CHAINCODE_POLICY=
TRANSIENT_DATA=
BLOCK_NUMBER=newest
EXPOSE_ENDPOINTS=true
CURRENT_ORG=org1.example.com
HOST_ADDRESSES=10.0.0.29
WORKING_DIRECTORY: .../Hyperledger
...
# Preparing for the following operations: *********************
verify options, download images, generate certificates, start network, network status, channel create, channel join, anchor update, profile generation, cc install, cc approve, cc commit, cc initialize, discover
.................
# Running operation: ******************************************
verify options
.
# Running operation: ******************************************
download images
............
# Running operation: ******************************************
generate certificates
.............
# Running operation: ******************************************
start network
.....................
# Running operation: ******************************************
network status
......
# Docker node status ******************************************
f8ff235a92_cli : Up Less than a second
primary-ca.org1.example.com : Up 1 second
orderer.org1.orderer-node.com : Up 2 seconds
internal.org1.example.com : Up 5 seconds
external.org1.example.com : Up 8 seconds
internal.org1.example.com.couchdb : Up 9 seconds
external.org1.example.com.couchdb : Up 10 seconds

Fabric network peer and orderer node health status **

external.org1.example.com internal.org1.example.com orderer.org1.orderer-node.com "OK"

Running operation: **

channel create ......

Running operation: **

channel join ...............

Running operation: **

anchor update .......

Running operation: **

profile generation .........................

Running operation: **

cc install .......

Run the chaincode install script on cli container ***

non-zero return code go: downloading github.com/hyperledger/fabric v1.4.1 go: finding module for package github.com/pkg/errors go: finding module for package github.com/spf13/viper go: finding module for package github.com/op/go-logging go: finding module for package google.golang.org/grpc go: finding module for package golang.org/x/net/context go: finding module for package golang.org/x/crypto/sha3 go: finding module for package google.golang.org/grpc/keepalive go: finding module for package google.golang.org/grpc/stats go: downloading github.com/golang/protobuf v1.3.1 go: downloading golang.org/x/crypto v0.0.0-20200820211705-5c72a883971a go: downloading google.golang.org/grpc v1.31.0 go: downloading golang.org/x/net v0.0.0-20200822124328-c89045814202 go: finding module for package github.com/sykesm/zap-logfmt go: finding module for package go.uber.org/zap/zapgrpc go: downloading github.com/pkg/errors v0.9.1 go: downloading go.uber.org/zap v1.15.0 go: downloading github.com/spf13/viper v1.7.1 go: downloading github.com/sykesm/zap-logfmt v0.0.3 go: finding module for package go.uber.org/zap go: finding module for package github.com/miekg/pkcs11 go: finding module for package go.uber.org/zap/zapcore go: finding module for package google.golang.org/grpc/peer go: finding module for package gopkg.in/yaml.v2 go: finding module for package github.com/grpc-ecosystem/go-grpc-middleware go: finding module for package google.golang.org/grpc/grpclog go: downloading github.com/op/go-logging v0.0.0-20160315200505-970db520ece7 go: finding module for package github.com/fsouza/go-dockerclient go: finding module for package google.golang.org/grpc/credentials go: finding module for package go.uber.org/zap/buffer go: finding module for package github.com/hyperledger/fabric-amcl/amcl go: downloading github.com/miekg/pkcs11 v1.0.3 go: downloading gopkg.in/yaml.v2 v2.3.0 go: finding module for package github.com/hyperledger/fabric-amcl/amcl/FP256BN go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.2.1 go: downloading github.com/fsouza/go-dockerclient v1.6.5 go: downloading github.com/hyperledger/fabric-amcl v0.0.0-20200424173818-327c9e2cf77a go: found github.com/op/go-logging in github.com/op/go-logging v0.0.0-20160315200505-970db520ece7 go: found github.com/pkg/errors in github.com/pkg/errors v0.9.1 go: found github.com/spf13/viper in github.com/spf13/viper v1.7.1 go: found google.golang.org/grpc in google.golang.org/grpc v1.31.0 go: found golang.org/x/net/context in golang.org/x/net v0.0.0-20200822124328-c89045814202 go: found github.com/grpc-ecosystem/go-grpc-middleware in github.com/grpc-ecosystem/go-grpc-middleware v1.2.1 go: found github.com/miekg/pkcs11 in github.com/miekg/pkcs11 v1.0.3 go: found go.uber.org/zap/zapcore in go.uber.org/zap v1.15.0 go: found golang.org/x/crypto/sha3 in golang.org/x/crypto v0.0.0-20200820211705-5c72a883971a go: found github.com/sykesm/zap-logfmt in github.com/sykesm/zap-logfmt v0.0.3 go: found gopkg.in/yaml.v2 in gopkg.in/yaml.v2 v2.3.0 go: found github.com/fsouza/go-dockerclient in github.com/fsouza/go-dockerclient v1.6.5 go: found github.com/hyperledger/fabric-amcl/amcl in github.com/hyperledger/fabric-amcl v0.0.0-20200424173818-327c9e2cf77a go: downloading github.com/golang/protobuf v1.3.3 go: downloading github.com/mitchellh/mapstructure v1.1.2 go: downloading github.com/spf13/jwalterweatherman v1.0.0 go: downloading github.com/fsnotify/fsnotify v1.4.7 go: downloading github.com/pelletier/go-toml v1.2.0 go: downloading github.com/spf13/pflag v1.0.3 go: downloading github.com/spf13/afero v1.1.2 go: downloading github.com/magiconair/properties v1.8.1 go: downloading gopkg.in/ini.v1 v1.51.0 go: downloading github.com/subosito/gotenv v1.2.0 go: downloading github.com/spf13/cast v1.3.0 go: downloading golang.org/x/text v0.3.2 go: downloading google.golang.org/genproto v0.0.0-20200423170343-7949de9c1215 go: downloading github.com/hashicorp/hcl v1.0.0 go: downloading golang.org/x/sys v0.0.0-20200420163511-1957bb5e6d1f go: downloading go.uber.org/atomic v1.6.0 go: downloading go.uber.org/multierr v1.5.0 go: downloading github.com/docker/docker v1.4.2-0.20191101170500-ac7306503d23 go: downloading github.com/Microsoft/go-winio v0.4.15-0.20200113171025-3fe6c5262873 go: downloading github.com/opencontainers/image-spec v1.0.1 go: downloading github.com/docker/go-units v0.4.0 go: downloading github.com/gogo/protobuf v1.3.1 go: downloading github.com/morikuni/aec v1.0.0 go: downloading github.com/Azure/go-ansiterm v0.0.0-20170929234023-d6e3b3328b78 go: downloading github.com/docker/distribution v2.7.1+incompatible go: downloading github.com/Microsoft/hcsshim v0.8.7 go: downloading github.com/opencontainers/go-digest v1.0.0-rc1 go: downloading github.com/docker/go-connections v0.4.0 go: downloading github.com/opencontainers/runc v0.1.1 go: downloading github.com/containerd/containerd v1.3.0 go: downloading github.com/containerd/continuity v0.0.0-20200228182428-0f16d7a0959c go: downloading github.com/sirupsen/logrus v1.4.2 go: downloading golang.org/x/sync v0.0.0-20190423024810-112230192c58 go: downloading github.com/konsorten/go-windows-terminal-sequences v1.0.1 Error: chaincode install failed with status: 500 - failed to invoke backing implementation of 'InstallChaincode': could not build chaincode: docker build failed: docker image inspection failed: Get "http://unix.sock/images/dev-org1.example.com- simple_1.0-b5c5a3d167303f6b7597dafb43cee2008531fcbf6c8c25f44a60dfcb29190ed4- 7e5a6e7149d525458e68003f43232d8c8078645ea26cecbe8ba85be6a04d4618/json": dial unix /host/var/run/docker.sock: connect: no such file or directory

STATS ***

minifab: ok=166 failed=1

real 1m53.054s user 0m52.911s sys 0m16.843s

litong01 commented 3 years ago

@edkaz looks like that your mac endpoint may not be at /var/run/docker.sock, can you check that?

litong01 commented 3 years ago

@edkaz I wonder if your mac and centos also have some ports blocked. take a look at the doc on centos in docs folder.

litong01 commented 3 years ago

@edkaz I've just installed the docker desktop on my mac, used exactly same command that you described above, everything worked without any issue at all. see below.

$ ./minifab up -o org1.example.com -s couchdb -c testchannel -e true
Minifab Execution Context:
    FABRIC_RELEASE=2.2.0
    CHANNEL_NAME=testchannel
    PEER_DATABASE_TYPE=couchdb
    CHAINCODE_LANGUAGE=go
    CHAINCODE_NAME=simple
    CHAINCODE_VERSION=1.0
    CHAINCODE_INIT_REQUIRED=true
    CHAINCODE_PARAMETERS="init","a","200","b","300"
    CHAINCODE_PRIVATE=false
    CHAINCODE_POLICY=
    TRANSIENT_DATA=
    BLOCK_NUMBER=newest
    EXPOSE_ENDPOINTS=true
    CURRENT_ORG=org1.example.com
    HOST_ADDRESSES=192.168.1.81
    WORKING_DIRECTORY: /Users/tongli/mywork
...
# Preparing for the following operations: *********************
  verify options, download images, generate certificates, start network, network status, channel create, channel join, anchor update, profile generation, cc install, cc approve, cc commit, cc initialize, discover
.................
# Running operation: ******************************************
  verify options
.
# Running operation: ******************************************
  download images
............
# Running operation: ******************************************
  generate certificates
.............
# Running operation: ******************************************
  start network
.....................
# Running operation: ******************************************
  network status
......
# Docker node status ******************************************
  5e4a59dfd5_cli : Up Less than a second
  ca1.org1.example.com : Up 1 second
  ca1.org0.example.com : Up 2 seconds
  orderer3.example.com : Up 3 seconds
  orderer2.example.com : Up 4 seconds
  orderer1.example.com : Up 5 seconds
  peer2.org1.example.com : Up 6 seconds
  peer1.org1.example.com : Up 7 seconds
  peer2.org0.example.com : Up 8 seconds
  peer1.org0.example.com : Up 10 seconds
  peer2.org1.example.com.couchdb : Up 11 seconds
  peer1.org1.example.com.couchdb : Up 12 seconds
  peer2.org0.example.com.couchdb : Up 13 seconds
  peer1.org0.example.com.couchdb : Up 13 seconds

# Fabric network peer and orderer node health status **********
  peer1.org0.example.com "OK"
  peer2.org0.example.com "OK"
  peer1.org1.example.com "OK"
  peer2.org1.example.com "OK"
  orderer1.example.com "OK"
  orderer2.example.com "OK"
  orderer3.example.com "OK"

# Running operation: ******************************************
  channel create
......
# Running operation: ******************************************
  channel join
...........................
# Running operation: ******************************************
  anchor update
............
# Running operation: ******************************************
  profile generation
.................................
# Running operation: ******************************************
  cc install
...........................
# Running operation: ******************************************
  cc approve
......
# Running operation: ******************************************
  cc commit
......
# Running operation: ******************************************
  cc initialize
......
# Running operation: ******************************************
  discover
........................

# STATS *******************************************************
minifab: ok=265 failed=0

real    3m34.598s
user    1m10.232s
sys 0m18.865s
litong01 commented 3 years ago

@edkaz I even did a blockquery after, things work as they should. Please check the things that I indicated above. In some cases, the chaincode install will take two tries due to network issues. You can try install command again and see if it works.

edkaz commented 3 years ago

@litong01 Hello Tong Li

I will check my end points. By the way, everything was working perfectly with the minifab before last Thursday PT so these errors are very recent. Minifab is a great tool and very very handy for working with remote servers.

From what I can diagnose the problem is that minifab is trying to install the default chaincode on a dev mode container but the dev mode containers are missing. Previously the dev containers were generated automatically but now I have just:

Edmunds-MBP:Hyperledger ed$ docker ps --format "{{.Names}}"

gives 

f8ff235a92_cli
primary-ca.org1.example.com
orderer.org1.example.com
internal.org1.example.com
external.org1.example.com
internal.org1.example.com.couchdb
external.org1.example.com.couchdb

But just to confirm for my Mac(s) that docker is installed and working:

ls -l /var/run/docker*
lrwxr-xr-x  1 root  daemon  67 24 Aug 07:57 /var/run/docker-cli.sock -> /Users/ed/Library/Containers/com.docker.docker/Data/docker-cli.sock
lrwxr-xr-x  1 root  daemon  63 24 Aug 07:57 /var/run/docker.sock -> /Users/ed/Library/Containers/com.docker.docker/Data/docker.sock

Again, the error thrown is in the in the last line of my previous post where minifab appears to be looking for /host/var/run/ while on my Mac its /var/run (see above). But again the problems for me only started last Thursday PT if that helps.

Best regards - Ed

litong01 commented 3 years ago

@edkaz you stated the following:

From what I can diagnose the problem is that minifab is trying to install the default chaincode on a dev mode container but the dev mode containers are missing.

But that is not true. minifab does not use any dev mode container nor there is a such thing called dev mode container. You may in the past see chaincode image named using a string started with dev, but it does not mean it is dev mode. There is no such thing called dev mode.

If you suspect that recent minifabric breaks, you can always fallback to an earlier version (you will have to extract an earlier commit and build it yourself) to see if it works. I do not think that is the case because recent changes have nothing to do with installing chaincode.

edkaz commented 3 years ago

@litong01

Thank you for getting back to me so quickly. Thank you also for your explanation regarding the naming of containers.

First, I reinstalled all containers and minifab on my Centos server and everything came up as it should . So all good there.

Second I may need to revert to an older version but for now the sequence of errors being generated on my Mac development machines are the following. Running minifab goes fine until... 

    # Run the chaincode install script on cli container ***********

gives

Error: chaincode install failed with status: 500 - failed to invoke backing implementation of 'InstallChaincode': could not build chaincode: docker build failed: docker image inspection failed: Get "http://unix.sock/images/dev-org1.example.com-simple_1.0-b5c5a3d167303f6b7597dafb43cee2008531fcbf6c8c25f44a60dfcb29190ed4-7e5a6e7149d525458e68003f43232d8c8078645ea26cecbe8ba85be6a04d4618/json": dial unix /host/var/run/docker.sock: connect: no such file or directory

If we break the error down seems to consist of a trace, which for now I'll just refer to as three errors. So, if I've read this correctly we have the following.

First Error : 

    Error: chaincode install failed with status: 500 - failed to invoke backing implementation of 'InstallChaincode': could not build 
    chaincode: docker build failed.

Second Error:

    docker image inspection failed: Get "http://unix.sock/images/dev-external.org1.example.com-simple_1.0-b5c5...4618/json

Third Error:

    dial unix /host/var/run/docker.sock: connect: no such file or directory

Working backwards, the third error is a bit surprising because at least on Darwin 19.6.0 and Centos 7.7, docker.sock is in the directory /var/run and not /host/var/run. It's surprising because minifab seems to resolve the Linux path correctly but does not resolve the Mac path correctly, unless of course I've missed something here.

For the second error, minifab prior to last Thursday PT generated containers with the prefix dev- on my Mac (I'm sorry about my earlier misunderstanding of the naming) but now does not. The installation code seems to be trying to access a non-existent container.

Now, by comparison on my Centos machine with the latest version of minifab downloaded earlier today I have the following containers:

docker ps --format "{{.Names}}"
dev-peer-2.org1.example.com-simple_1.0-b5c5a3d167303f6b7597dafb43cee2008531fcbf6c8c25f44a60dfcb29190ed4
dev-peer-3.org1.example.com-simple_1.0-b5c5a3d167303f6b7597dafb43cee2008531fcbf6c8c25f44a60dfcb29190ed4
dev-peer-4.org1.example.com-simple_1.0-b5c5a3d167303f6b7597dafb43cee2008531fcbf6c8c25f44a60dfcb29190ed4
dev-peer-1.org1.example.com-simple_1.0-b5c5a3d167303f6b7597dafb43cee2008531fcbf6c8c25f44a60dfcb29190ed4
c37b52d45d_cli
ca-secondary.org1.example.com
orderer-1.order-org.example.com
peer-4.org1.example.com
peer-3.org1.example.com
peer-2.org1.example.com
peer-1.org1.example.com
peer-4.org1.example.com.couchdb
peer-3.org1.example.comcouchdb
peer-2.org1.example.com.couchdb
peer-1.org1.example.com.couchdb

and everything works perfectly. So this seems to be a Darwin/MacOS issue.

Finally the first error is only present on my Darwin system and not on the CentOS system but I'm assuming it's the result of the second and third errors. Hope this helps.

Regards - Ed

litong01 commented 3 years ago

@edkaz I suggest that you cleanup your system on mac. and get the latest code, and also check your mac firewall etc. cc install can fail most due to network issues and timeout since it needs to pull down a lot of things. when cc install fails, you can try the 2nd time and see if it is ok.

edkaz commented 3 years ago

@litong01 Hello and thanks again for the quick reply. I have reinstalled all containers and minifab but haven't tried the repeating my CC install so will try that and let you know how we go.

Regards - Ed

litong01 commented 3 years ago

@edkaz any updates on this one? I am trying to close out some old issues which no longer valid.

edkaz commented 3 years ago

@litong01 Sorry no luck so far but I'll close it out for now. Regards - Ed

litong01 commented 3 years ago

@edkaz I actually would like to get it resolved rather than simply close it since it is still an issue in your env.

litong01 commented 3 years ago

No activity for long time. Close the issue.

sebastianrothe commented 3 years ago

I have the exact same issue with Docker Desktop 3.0.3. It works once I disable gRPC-FUSE file sharing in the settings from Docker Desktop (Preferences -> Experimental Features. Don't know what the problem is, though.