search

hyperledger / cello

Operating System for Enterprise Blockchain
https://wiki.hyperledger.org/display/cello
Apache License 2.0
886 stars 439 forks source link

Instantiate the chaincode error #207

Closed victorming closed 3 years ago

victorming commented 3 years ago

I deployed ok the channel. And when I was testing the 'chaincode_example02.go', the system hanged after clicking the submit button: image The logs of user-dashboard instance: image ChannelEventHub has been shutdown.

victorming commented 3 years ago

I checked the logs of peer containers, following are logs of one of the peer container:

2021-01-22 06:52:06.885 UTC [core.comm] ServerHandshake -> ERRO 0fc TLS handshake failed with error EOF server=PeerServer remoteaddress=192.168.101.243:34260 2021-01-22 06:54:07.047 UTC [core.comm] ServerHandshake -> ERRO 0fd TLS handshake failed with error EOF server=PeerServer remoteaddress=192.168.101.243:34444 2021-01-22 06:56:07.214 UTC [core.comm] ServerHandshake -> ERRO 0fe TLS handshake failed with error EOF server=PeerServer remoteaddress=192.168.101.243:34608 2021-01-22 06:58:07.388 UTC [core.comm] ServerHandshake -> ERRO 0ff TLS handshake failed with error EOF server=PeerServer remoteaddress=192.168.101.243:34774 2021-01-22 07:00:07.556 UTC [core.comm] ServerHandshake -> ERRO 100 TLS handshake failed with error EOF server=PeerServer remoteaddress=192.168.101.243:34964 2021-01-22 07:00:51.048 UTC [endorser] callChaincode -> INFO 101 [powerpole-channel][c54e4383] Entry chaincode: name:"lscc" 2021-01-22 07:00:51.060 UTC [endorser] callChaincode -> INFO 102 [powerpole-channel][c54e4383] Exit chaincode: name:"lscc" (12ms) 2021-01-22 07:00:51.060 UTC [comm.grpc.server] 1 -> INFO 103 unary call completed grpc.service=protos.Endorser grpc.method=ProcessProposal grpc.peer_address=192.168.101.243:51560 grpc.code=OK grpc.call_duration=13.730573ms 2021-01-22 07:02:07.701 UTC [core.comm] ServerHandshake -> ERRO 104 TLS handshake failed with error EOF server=PeerServer remoteaddress=192.168.101.243:35176 2021-01-22 07:02:50.538 UTC [endorser] callChaincode -> INFO 105 [powerpole-channel][db3be8f5] Entry chaincode: name:"lscc" 2021-01-22 07:02:50.552 UTC [endorser] callChaincode -> INFO 106 [powerpole-channel][db3be8f5] Exit chaincode: name:"lscc" (14ms) 2021-01-22 07:02:50.552 UTC [comm.grpc.server] 1 -> INFO 107 unary call completed grpc.service=protos.Endorser grpc.method=ProcessProposal grpc.peer_address=192.168.101.243:51746 grpc.code=OK grpc.call_duration=15.235301ms 2021-01-22 07:02:51.290 UTC [comm.grpc.server] 1 -> INFO 108 streaming call completed grpc.service=protos.Deliver grpc.method=DeliverFiltered grpc.peer_address=192.168.101.243:51560 error="context finished before block retrieved: context canceled" grpc.code=Unknown grpc.call_duration=2m0.130955326s 2021-01-22 07:04:07.871 UTC [core.comm] ServerHandshake -> ERRO 109 TLS handshake failed with error EOF server=PeerServer remoteaddress=192.168.101.243:35368

where 192.168.101.243 is where the operator-dashboard and user-dashboard containers are deployed.

victorming commented 3 years ago

The TLS handshake failures might not be the cause for this chaincode instatiation error. I recreated the network and checked the logs of orderer container:

2021-01-22 07:23:23.251 UTC [orderer.common.server] Start -> INFO 017 Starting orderer: Version: 1.4.2 Commit SHA: c6cc550 Go version: go1.11.5 OS/Arch: linux/amd64 2021-01-22 07:23:23.251 UTC [orderer.common.server] Start -> INFO 018 Beginning to serve requests 2021-01-22 07:23:23.251 UTC [orderer.consensus.etcdraft] run -> INFO 019 This node is picked to start campaign channel=testchainid node=3 2021-01-22 07:23:23.282 UTC [orderer.consensus.etcdraft] apply -> INFO 01a Applied config change to add node 1, current nodes in channel: [1 2 3] channel=testchainid node=3 2021-01-22 07:23:23.282 UTC [orderer.consensus.etcdraft] apply -> INFO 01b Applied config change to add node 2, current nodes in channel: [1 2 3] channel=testchainid node=3 2021-01-22 07:23:23.282 UTC [orderer.consensus.etcdraft] apply -> INFO 01c Applied config change to add node 3, current nodes in channel: [1 2 3] channel=testchainid node=3 2021-01-22 07:23:24.451 UTC [orderer.consensus.etcdraft] Step -> INFO 01d 3 is starting a new election at term 1 channel=testchainid node=3 2021-01-22 07:23:24.451 UTC [orderer.consensus.etcdraft] becomePreCandidate -> INFO 01e 3 became pre-candidate at term 1 channel=testchainid node=3 2021-01-22 07:23:24.452 UTC [orderer.consensus.etcdraft] poll -> INFO 01f 3 received MsgPreVoteResp from 3 at term 1 channel=testchainid node=3 2021-01-22 07:23:24.452 UTC [orderer.consensus.etcdraft] campaign -> INFO 020 3 [logterm: 1, index: 3] sent MsgPreVote request to 1 at term 1 channel=testchainid node=3 2021-01-22 07:23:24.452 UTC [orderer.consensus.etcdraft] campaign -> INFO 021 3 [logterm: 1, index: 3] sent MsgPreVote request to 2 at term 1 channel=testchainid node=3 2021-01-22 07:23:24.456 UTC [orderer.consensus.etcdraft] poll -> INFO 022 3 received MsgPreVoteResp from 2 at term 1 channel=testchainid node=3 2021-01-22 07:23:24.456 UTC [orderer.consensus.etcdraft] stepCandidate -> INFO 023 3 [quorum:2] has received 2 MsgPreVoteResp votes and 0 vote rejections channel=testchainid node=3 2021-01-22 07:23:24.456 UTC [orderer.consensus.etcdraft] becomeCandidate -> INFO 024 3 became candidate at term 2 channel=testchainid node=3 2021-01-22 07:23:24.456 UTC [orderer.consensus.etcdraft] poll -> INFO 025 3 received MsgVoteResp from 3 at term 2 channel=testchainid node=3 2021-01-22 07:23:24.457 UTC [orderer.consensus.etcdraft] campaign -> INFO 026 3 [logterm: 1, index: 3] sent MsgVote request to 2 at term 2 channel=testchainid node=3 2021-01-22 07:23:24.457 UTC [orderer.consensus.etcdraft] campaign -> INFO 027 3 [logterm: 1, index: 3] sent MsgVote request to 1 at term 2 channel=testchainid node=3 2021-01-22 07:23:24.615 UTC [orderer.consensus.etcdraft] poll -> INFO 028 3 received MsgVoteResp from 2 at term 2 channel=testchainid node=3 2021-01-22 07:23:24.616 UTC [orderer.consensus.etcdraft] stepCandidate -> INFO 029 3 [quorum:2] has received 2 MsgVoteResp votes and 0 vote rejections channel=testchainid node=3 2021-01-22 07:23:24.616 UTC [orderer.consensus.etcdraft] becomeLeader -> INFO 02a 3 became leader at term 2 channel=testchainid node=3 2021-01-22 07:23:24.616 UTC [orderer.consensus.etcdraft] run -> INFO 02b raft.node: 3 elected leader 3 at term 2 channel=testchainid node=3 2021-01-22 07:23:24.642 UTC [orderer.consensus.etcdraft] serveRequest -> INFO 02c Raft leader changed: 0 -> 3 channel=testchainid node=3 2021-01-22 07:23:24.642 UTC [orderer.consensus.etcdraft] run -> INFO 02d Leader 3 is present, quit campaign channel=testchainid node=3 2021-01-22 07:23:24.656 UTC [orderer.consensus.etcdraft] serveRequest -> INFO 02e Start accepting requests as Raft leader at block [0] channel=testchainid node=3 2021-01-22 07:24:19.225 UTC [core.comm] ServerHandshake -> ERRO 02f TLS handshake failed with error EOF server=Orderer remoteaddress=192.168.101.243:50106 2021-01-22 07:26:19.393 UTC [core.comm] ServerHandshake -> ERRO 030 TLS handshake failed with error EOF server=Orderer remoteaddress=192.168.101.243:50268 2021-01-22 07:26:48.806 UTC [core.comm] ServerHandshake -> ERRO 031 TLS handshake failed with error EOF server=Orderer remoteaddress=192.168.101.243:50310

I am confused that the orderer node keeps connectting to operator-dashboard. I suppose the network should not be involved to the baas platform after creation. Anybody can help?

victorming commented 3 years ago

After several times to recreate the Organizations, Networks and install the chaincode, I figured out the issue might happened as the peer nodes failed to connect the orderer nodes. Following is the logs of peer node:

2021-01-22 09:16:23.784 UTC [ConnProducer] NewConnection -> ERRO 081 Failed connecting to {orderer2-osp:30002 [OspMSP]} , error: context deadline exceeded 2021-01-22 09:16:26.786 UTC [ConnProducer] NewConnection -> ERRO 082 Failed connecting to {orderer1-osp:30001 [OspMSP]} , error: context deadline exceeded 2021-01-22 09:16:29.787 UTC [ConnProducer] NewConnection -> ERRO 083 Failed connecting to {orderer0-osp:30000 [OspMSP]} , error: context deadline exceeded 2021-01-22 09:16:29.787 UTC [ConnProducer] NewConnection -> ERRO 084 Could not connect to any of the endpoints: [{orderer2-osp:30002 [OspMSP]} {orderer1-osp:30001 [OspMSP]} {orderer0-osp:30000 [OspMSP]}] 2021-01-22 09:16:29.787 UTC [deliveryClient] connect -> ERRO 085 Failed obtaining connection: could not connect to any of the endpoints: [{orderer2-osp:30002 [OspMSP]} {orderer1-osp:30001 [OspMSP]} {orderer0-osp:30000 [OspMSP]}] 2021-01-22 09:16:29.787 UTC [deliveryClient] try -> WARN 086 Got error: could not connect to any of the endpoints: [{orderer2-osp:30002 [OspMSP]} {orderer1-osp:30001 [OspMSP]} {orderer0-osp:30000 [OspMSP]}] , at 10 attempt. Retrying in 8m32s

I put the orderer0-osp in the /etc/hosts but in vain. Why would the peer nodes report connection failure to order?

victorming commented 3 years ago

Only when the chaincode is instantiated, the peer node would try to connect to the orderer. So everything is OK until instantiating the chaincode. I use the Fabric v1.4.2. Anybody encountered the same issue?

victorming commented 3 years ago

I tried the peer commond to instantiate the cc, following is the cmd and output:

2021-01-22 09:30:55.025 UTC [main] InitCmd -> WARN 001 CORE_LOGGING_LEVEL is no longer supported, please use the FABRIC_LOGGING_SPEC environment variable 2021-01-22 09:30:55.043 UTC [main] SetOrdererEnv -> WARN 002 CORE_LOGGING_LEVEL is no longer supported, please use the FABRIC_LOGGING_SPEC environment variable Error: error getting broadcast client: orderer client failed to connect to orderer0.zonesion.org:7050: failed to create new connection: connection error: desc = "transport: error while dialing: dial tcp: lookup orderer0.zonesion.org on 127.0.0.11:53: no such host"

It's clear the peer node can't connect to orderer0. But why? Following is the network's topology: image Everything seems OK.

victorming commented 3 years ago

Any guys established the real fabric network using this so-called BAAS platform?

XuHugo commented 3 years ago

I can't see your picture，What kind of host do you use？ In addition, pay attention to whether the time of the whole system is synchronized。

victorming commented 3 years ago

I use docker as the host. And I checked the README-H3C.md, they said: " If the host of the orderer node selected a docker type host, only the consensus type of solo mode can be supported.". I selected "etcd-raft" as the consensus type. Is it possible this caused the issue?

XuHugo commented 3 years ago

You should listen to him，have a try。

victorming commented 3 years ago

Still not work. I recreated the organizations and the network using SOLO instead of etcd-raft. When the network is deployed, the orderer node show errors:

2021-01-25 03:37:37.857 UTC [core.comm] ServerHandshake -> ERRO 009 TLS handshake failed with error EOF server=Orderer remoteaddress=192.168.101.243:41676 2021-01-25 03:39:38.005 UTC [core.comm] ServerHandshake -> ERRO 00a TLS handshake failed with error EOF server=Orderer remoteaddress=192.168.101.243:41852

192.168.101.243 is the master node where operator-dashboard and user-dashboard containers are deployed.

In spit of above errors, the operator-dashboard shows the network is created and running SUCCESSFULLY. The I proceed into following steps to create the channel and install the chaincode. After install the chaincode, the peer node show errors:

2021-01-25 03:44:59.869 UTC [ConnProducer] NewConnection -> ERRO 067 Failed connecting to {orderer-osp:30000 [OspMSP]} , error: context deadline exceeded 2021-01-25 03:44:59.869 UTC [ConnProducer] NewConnection -> ERRO 068 Could not connect to any of the endpoints: [{orderer-osp:30000 [OspMSP]}] 2021-01-25 03:44:59.869 UTC [deliveryClient] connect -> ERRO 069 Failed obtaining connection: could not connect to any of the endpoints: [{orderer-osp:30000 [OspMSP]}] 2021-01-25 03:44:59.869 UTC [deliveryClient] try -> WARN 06a Got error: could not connect to any of the endpoints: [{orderer-osp:30000 [OspMSP]}] , at 9 attempt. Retrying in 4m16s

However the operator-dashboard still respond chaincode install is OK. Then I tried to instantiate the chaincode, the operator-dashboard button keeps rolling and no response anymore. Clearly the instantiation of chaincode is failed.

I checked the crpyto-config.yaml and configtx.yaml generated by the operator-dashboard and noticed OrdererOrg and PeerOrg all use SAN as the hostname. And every docker node reports bad connections to the SAN addresses. Is it possible the SANs caused this issue? And can't I delete the SAN configuration in the crpyto-config.yaml and configtx.yaml to overcome this issue?

Following is the crypto-config.yaml:

OrdererOrgs:

  • CA: {Country: '', Locality: '', Province: ''} Domain: zonesion.org Name: Osp Specs:
    • Hostname: orderer SANS: [orderer-osp]

PeerOrgs:

  • CA: {Country: "\u4E2D\u56FD", Locality: "\u6B66\u6C49", Province: "\u6E56\u5317"} Domain: ppso0.zonesion.org EnableNodeOUs: true Name: Ppso0 Specs:
    • Hostname: peer0 SANS: [peer0-ppso0]
  • CA: {Country: "\u4E2D\u56FD", Locality: "\u6B66\u6C49", Province: "\u6E56\u5317"} Domain: ppso1.zonesion.org EnableNodeOUs: true Name: Ppso1 Specs:
    • Hostname: peer0 SANS: [peer0-ppso1]
  • CA: {Country: "\u4E2D\u56FD", Locality: "\u6B66\u6C49", Province: "\u6E56\u5317"} Domain: ppso2.zonesion.org EnableNodeOUs: true Name: Ppso2 Specs:
    • Hostname: peer0 SANS: [peer0-ppso2]
victorming commented 3 years ago

The first error doesn't likely cause any trouble since it indicates the master node cli can't connect the fabric network:

2021-01-25 03:37:37.857 UTC [core.comm] ServerHandshake -> ERRO 009 TLS handshake failed with error EOF server=Orderer remoteaddress=192.168.101.243:41676

But the second error is deadly critical:

2021-01-25 03:44:59.869 UTC [ConnProducer] NewConnection -> ERRO 067 Failed connecting to {orderer-osp:30000 [OspMSP]} , error: context deadline exceeded

I checked the network used by orderer node and peer nodes. The ip address of orderer node is not consistent with peer nodes: ---> Orderer network: docker network inspect bb2bc2a1831f_celloNet

[ { "Name": "bb2bc2a1831f_celloNet", "Id": "b89800cecacb76a4d24fd5e43a5f56a9895e2b7837b491ef3f04595e49cf5e8d", "Created": "2021-01-25T14:05:37.498744984+08:00", "Scope": "local", "Driver": "bridge", "EnableIPv6": false, "IPAM": { "Driver": "default", "Options": null, "Config": [ { "Subnet": "192.168.48.0/20", "Gateway": "192.168.48.1" } ] }, "Internal": false, "Attachable": true, "Ingress": false, "ConfigFrom": { "Network": "" }, "ConfigOnly": false, "Containers": { "1a1971cd667cff50c726340c1ac6e6657b2798b1dd5f6dde42950f3463c9fe84": { "Name": "bb2bc2a1831f_orderer-osp", "EndpointID": "c55adaf187d8f500964de53c15dd276905110947e6ebbd79e992ba2124235b5d", "MacAddress": "02:42:c0:a8:30:02", "IPv4Address": "192.168.48.2/20", "IPv6Address": "" } }, "Options": {}, "Labels": { "com.docker.compose.network": "celloNet", "com.docker.compose.project": "bb2bc2a1831f" } } ]

---> Peer node: docker network inspect bb2bc2a1831f_celloNet

[ { "Name": "bb2bc2a1831f_celloNet", "Id": "5a6bf660f4a7366e9ea79d2b2d6a3ad7e9fdebff0904533a1b933772d6e7c8c7", "Created": "2021-01-25T14:05:48.906994798+08:00", "Scope": "local", "Driver": "bridge", "EnableIPv6": false, "IPAM": { "Driver": "default", "Options": null, "Config": [ { "Subnet": "172.31.0.0/16", "Gateway": "172.31.0.1" } ] }, "Internal": false, "Attachable": true, "Ingress": false, "ConfigFrom": { "Network": "" }, "ConfigOnly": false, "Containers": { "0a07e72000f50a1a45fd565b6c779f851b399d479c92032c92dd302383fd5e4d": { "Name": "bb2bc2a1831f_ca.ppso1.zonesion.org", "EndpointID": "5b23421cebd3215591c32bedc19741ee74f316076820bc7611b590c56974a026", "MacAddress": "02:42:ac:1f:00:02", "IPv4Address": "172.31.0.2/16", "IPv6Address": "" }, "1780d3a770c0dfa5f30891138baf94e83e7c9d2ac79fb1d2030a269c44389307": { "Name": "bb2bc2a1831f_peer0.ppso1.zonesion.org", "EndpointID": "4204d904edac2ec848db59785fb6ad23d5832124bcded095da1e6f496fb84b08", "MacAddress": "02:42:ac:1f:00:03", "IPv4Address": "172.31.0.3/16", "IPv6Address": "" }, "52d2281801ccf667b6fd9f279c6e90ecfc363233a162e09583efdd5d7a780990": { "Name": "dev-peer0.ppso1.zonesion.org-testChainCode-1.0", "EndpointID": "47b76ad716f899df36975513f7404f46ed81da5db92515b5ca265754f87db124", "MacAddress": "02:42:ac:1f:00:05", "IPv4Address": "172.31.0.5/16", "IPv6Address": "" }, "697bf0c4d8df2f12e47b58827735f2732b3a511087eedd24b2478b3a7ce954bc": { "Name": "bb2bc2a1831f_couchdb.peer0.ppso1.zonesion.org", "EndpointID": "dfbc567facccbdf04cd8704123fa91ceee29128fc7edd20ddb3f9ef1e6d6ebff", "MacAddress": "02:42:ac:1f:00:04", "IPv4Address": "172.31.0.4/16", "IPv6Address": "" } }, "Options": {}, "Labels": { "com.docker.compose.network": "celloNet", "com.docker.compose.project": "bb2bc2a1831f" } } ]

As above shows, the orderer node's network is 192.168.48.0/20, while peer node's network is 172.31.0.0/16. So the peer can't connect to orderer and failed to instantiate the chaincode. Is this the root cause? @XuHugo

XuHugo commented 3 years ago

Yes, Orderer and peer should be in the same network segment；

2021年1月25日 下午12:04，victorming notifications@github.com 写道：

192.168.101.243

victorming commented 3 years ago

But why the networks of Orderer and Peer are so different even with the same docker network service -- "bb2bc2a1831f_celloNet". Is this a bug?

XuHugo commented 3 years ago

Are peer and orderer started on the same host？

victorming commented 3 years ago

No, peer and orderer run on different hosts. Till now only one network like byfn where all orderer and peer containers run on one host is successfully deployed through this baas. This is meaningless in production scenario since we want to deploy fabric network on docker swarm or k8s cluster.

XuHugo commented 3 years ago

I can't figure out what the problem is. I may need to actually look at the environment.

2021年1月26日 下午4:07，victorming notifications@github.com 写道：

No, peer and orderer run on different hosts. Till now only one network like byfn where all orderer and peer containers run on one host is successfully deployed through this baas.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hyperledger/cello/issues/207#issuecomment-767374531, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADIBSP6A2Y4GC77TQBUUGM3S3ZZ3PANCNFSM4WOA5CEA.

victorming commented 3 years ago

The key point is that, when orderer and peers are deployed on different hosts, orderer container's network and peer container's network are completely different even though they share the same docker network id: orderer ==>192.168.48.0/20 while peer nodes network ==>172.31.0.0/16. So when instantiating the chaincode the peer nodes report failed to connect to orderer. I'm checking the code and try to figure out the root cause. If you have successfully deployed any multiple-organizations fabric network on docker swarm or k8s, please give us a link for the operational guide. Many thinks!!

XuHugo commented 3 years ago

Cello already supports k8s。If you want to deploy the same network on different hosts, you need to solve the network problems yourself.

2021年1月26日 下午5:27，victorming notifications@github.com 写道：

The key point is that, when orderer and peers are deployed on different hosts, orderer container's network and peer container's network are completely different even though they share the same docker network id: orderer ==>192.168.48.0/20 while peer nodes network ==>172.31.0.0/16. So when instantiating the chaincode the peer nodes report failed to connect to orderer. I'm checking the code and try to figure out the root cause. If you have successfully deployed any multiple-organizations fabric network on docker swarm or k8s, please give us a link for the operational guide. Many thinks!!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hyperledger/cello/issues/207#issuecomment-767416265, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADIBSPZC6TIGJ4SVVGGTFZDS32DI5ANCNFSM4WOA5CEA.

victorming commented 3 years ago

Got it! Maybe I should turn to k8s directly. Thank you anyway!