search

hyperledger / caliper

A blockchain benchmark framework to measure performance of multiple blockchain solutions https://wiki.hyperledger.org/display/caliper
https://hyperledger.github.io/caliper/
Apache License 2.0
650 stars 402 forks source link

Caliper Multi-Host Timeout error #510

Closed nimaafraz closed 5 years ago

nimaafraz commented 5 years ago

I'm trying to run a simple benchmark with 2org1peergoleveldb network with the peers of the second Org on a separate VM. However, I get a timeout Error when initializing the simple benchmark. I only change the IPs of the peers in the fabric-ccp-go.yaml to the external IP. I made sure that the ports are open in the second VM. I wonder if there are more steps to take for the multi-host caliper.

Actual Behavior

node ./scripts/main.js "-c" "benchmark/simple/config-feedback-rate.yaml" "-n" "network/fabric-v1.1/2org1peergoleveldb/fabric-ccp-go.yaml"

info: [bench-flow.js]: ####### Caliper Test ####### Creating orderer.example.com ... done Creating network "2org1peergoleveldb_default" with the default driver Creating peer0.org1.example.com ... done Creating ca.org1.example.com ... Creating orderer.example.com ... Creating peer0.org2.example.com ... Creating peer0.org1.example.com ... info: [adapters/fabric-ccp]: Fabric SDK version: 1.1.0; TLS: none; Fabric v1.0 compatibility mode info: [adapters/fabric-ccp]: Org1's registrar enrolled successfully info: [adapters/fabric-ccp]: Org2's registrar enrolled successfully info: [adapters/fabric-ccp]: Org1's admin's materials are successfully loaded info: [adapters/fabric-ccp]: Org2's admin's materials are successfully loaded info: [adapters/fabric-ccp]: client0.org1.example.com's materials are successfully loaded info: [adapters/fabric-ccp]: client0.org2.example.com's materials are successfully loaded info: [adapters/fabric-ccp]: mychannel successfully created info: [adapters/fabric-ccp]: Sleeping 5s... info: [adapters/fabric-ccp]: Org1's peers successfully joined mychannel: peer0.org1.example.com info: [adapters/fabric-ccp]: peer0.org2.example.com has already joined mychannel info: [adapters/fabric-ccp]: Sleeping 5s... info: [adapters/fabric-ccp]: Installing chaincodes for mychannel... info: [adapters/fabric-ccp]: simple@v0 is already installed on peer0.org2.example.com info: [packager/Golang.js]: packaging GOLANG from contract/fabric/simple/go info: [adapters/fabric-ccp]: simple@v0 successfully installed on Org1's peers: peer0.org1.example.com info: [adapters/fabric-ccp]: Instantiating simple@v0 in mychannel. This might take some time... error: [adapters/fabric-ccp]: The following errors occured while instantiating simple@v0 in mychannel:

#######################################

Test summary: 0 succeeded, 0 failed

#######################################

info: [scripts/main.js]: Benchmark run successfully

## Possible Fix I tried to increase the timeout values in /caliper-nima/src/adapters/fabric-ccp/fabric.js as follow but it did not make any difference. Still the same timeout error.

    this.configChaincodeInstantiateTimeout = config.get('fabricCcp:timeout:chaincodeInstantiate', 30000000);
    this.configChaincodeInstantiateEventTimeout = config.get('fabricCcp:timeout:chaincodeInstantiateEvent', 30000000);
    this.configDefaultTimeout = config.get('fabricCcp:timeout:invokeOrQuery', 6000000);

Steps to Reproduce

  1. Change the IPs in the fabric-ccp-go.yaml (the link for the file https://justpaste.it/2bn4f)
  2. npm run bench -- -c benchmark/simple/config-feedback-rate.yaml -n network/fabric-v1.1/2org1peergoleveldb/fabric-ccp-go.yaml

Your Environment

Google cloud VMs Ubuntu 19.04 custom (1 vCPU, 7.5 GB memory) Caliper "git checkout 4156c4da7105fd1c2b848573a9943bfc9900becb"

aklenik commented 5 years ago

@nimaafraz Please use an up2date version of Caliper. Moreover, the peers also have a setting for increasing the chaincode instantiation timeout. For the client-side timeout configs, see this docs: https://hyperledger.github.io/caliper/docs/Runtime_Configuration.html

nimaafraz commented 5 years ago

@nimaafraz Please use an up2date version of Caliper. Moreover, the peers also have a setting for increasing the chaincode instantiation timeout. For the client-side timeout configs, see this docs: https://hyperledger.github.io/caliper/docs/Runtime_Configuration.html

@aklenik Thanks. I upgraded to the latest version. Also I increased the client-side timeout values. Now it does not give me a timeout error but it gets stuck in the following line for hours (depending on the timeout value): info: [adapters/fabric-ccp]: Instantiating simple@v0 in mychannel. This might take some time...

However, when looking at the second VM's Docker status, I can see that when I run the benchmark in the first VM, a new 'dev-peer0.org2.example.com-simple-v0' container is created in the second machine. I think the problem could be that the first VM cannot access the peer0 on the second VM. Do I have to maybe expose the docker containers for peers etc externally?

aklenik commented 5 years ago

@nimaafraz How is your network distributed? Every Fabric component (peer and orderer) endpoint should be exposed to Caliper (and each other). Also, take a look at the peer logs, might have some instantiation errors.

nimaafraz commented 5 years ago

@aklenik I have two VMs where I run the Org1's peer and CA and the single orderer in VM1, and Org2's peer and CA on VM2. Here is the network config file: https://justpaste.it/2bn4f I first make sure that the local version of the benchmark runs on both of the VMs. Then on VM2 I run: "docker-compose -f network/fabric-v1.4/2org1peergoleveldb/docker-compose.yaml up -d;sleep 3s" so the containers would start. Then on the first machine, I run the benchmark and get the timeout error. I checked the peer0.0rg1 logs and here is what I see: 2019-07-17 11:16:05.806 UTC [lscc] executeInstall -> INFO 030 Installed Chaincode [simple] Version [v0] to peer 2019-07-17 11:16:06.677 UTC [blocksProvider] DeliverBlocks -> WARN 031 [mychannel] Got error &{NOT_FOUND} 2019-07-17 11:16:06.778 UTC [ConnProducer] DisableEndpoint -> WARN 032 Only 1 endpoint remained, will not black-list it 2019-07-17 11:16:06.781 UTC [blocksProvider] DeliverBlocks -> WARN 033 [mychannel] Got error &{NOT_FOUND}

and my orderer logs on VM2:

2019-07-17 12:20:30.823 UTC [common/deliver] deliverBlocks -> DEBU d0a Rejecting deliver for 172.19.0.5:47102 because channel mychannel not found 2019-07-17 12:20:30.823 UTC [common/deliver] Handle -> DEBU d0b Waiting for new SeekInfo from 172.19.0.5:47102 2019-07-17 12:20:30.823 UTC [common/deliver] Handle -> DEBU d0c Attempting to read seek info message from 172.19.0.5:47102 2019-07-17 12:20:40.824 UTC [common/deliver] Handle -> WARN d0d Error reading from 172.19.0.5:47102: rpc error: code = Canceled desc = context canceled 2019-07-17 12:20:40.824 UTC [orderer/common/server] func1 -> DEBU d0e Closing Deliver stream 2019-07-17 12:20:40.825 UTC [orderer/common/server] Deliver -> DEBU d0f Starting new Deliver handler 2019-07-17 12:20:40.825 UTC [common/deliver] Handle -> DEBU d10 Starting new deliver loop for 172.19.0.5:47104 2019-07-17 12:20:40.825 UTC [common/deliver] Handle -> DEBU d11 Attempting to read seek info message from 172.19.0.5:47104 2019-07-17 12:20:40.826 UTC [common/deliver] deliverBlocks -> DEBU d12 Rejecting deliver for 172.19.0.5:47104 because channel mychannel not found 2019-07-17 12:20:40.826 UTC [common/deliver] Handle -> DEBU d13 Waiting for new SeekInfo from 172.19.0.5:47104 2019-07-17 12:20:40.826 UTC [common/deliver] Handle -> DEBU d14 Attempting to read seek info message from 172.19.0.5:47104 2019-07-17 12:20:50.827 UTC [common/deliver] Handle -> WARN d15 Error reading from 172.19.0.5:47104: rpc error: code = Canceled desc = context canceled

nimaafraz commented 5 years ago

I'm still struggling with this. What I did further was to make sure the contents of "caliper/network/fabric-v1.x/config/crypto-config" is the same in the two VMs. I'm not sure if this is necessary or not but anyway, I get another error now:

npm run bench -- -c benchmark/simple/config-linear-rate.yaml -n network/fabric-v1.4/2org1peergoleveldb/fabric-node.json

caliper@0.1.0 bench /home/nima/caliper node ./scripts/main.js "-c" "benchmark/simple/config-linear-rate.yaml" "-n" "network/fabric-v1.4/2org1peergoleveldb/fabric-node.json"

info: [bench-flow.js]: ####### Caliper Test ####### info: [bench-flow.js]: Executing command: docker-compose -f network/fabric-v1.4/2org1peergoleveldb/docker-compose.yaml up -d;sleep 3s The Docker Engine you're using is running in swarm mode.

Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.

To deploy your application across the swarm, use docker stack deploy. Creating orderer.example.com ... done Creating network "2org1peergoleveldb_default" with the default driver Creating peer0.org1.example.com ... done Creating ca.org2.example.com ... Creating orderer.example.com ... Creating peer0.org1.example.com ... Creating peer0.org2.example.com ... info: [create-channel.js]: Creating mychannel... info: [create-channel.js]: Created mychannel successfully info: [create-channel.js]: Sleeping 5s... info: [join-channel.js]: Joining channels... info: [join-channel.js]: Joining organization org1 to channel mychannel... info: [join-channel.js]: Successfully joined peerOrg1's peers to mychannel info: [join-channel.js]: Joining organization org2 to channel mychannel... info: [join-channel.js]: Successfully joined peerOrg2's peers to mychannel info: [join-channel.js]: Successfully joined mychannel info: [install-chaincode.js]: installing all chaincodes...... info: [install-chaincode.js]: Installing chaincode marbles... info: [install-chaincode.js]: Installing chaincode marbles... info: [install-chaincode.js]: Installed chaincode marbles successfully in all peers info: [install-chaincode.js]: Installing chaincode simple... info: [install-chaincode.js]: Installing chaincode simple... info: [install-chaincode.js]: Installed chaincode simple successfully in all peers info: [instantiate-chaincode.js]: Instantiating chaincodes... info: [e2eUtils.js]: instantiate proposal was good info: [e2eUtils.js]: instantiate proposal was good info: [e2eUtils.js]: Successfully sent Proposal and received ProposalResponse info: [e2eUtils.js]: The chaincode instantiate transaction has been committed on peer 10.142.0.6:7051 info: [e2eUtils.js]: The chaincode instantiate transaction was valid. error: [fabric.js]: Fabric chaincode install failed: TypeError: Cannot read property 'stack' of undefined at Object.run (/home/nima/caliper/src/adapters/fabric/instantiate-chaincode.js:46:68) at error: [bench-flow.js]: Error: TypeError: Cannot read property 'stack' of undefined at Object.run (/home/nima/caliper/src/adapters/fabric/instantiate-chaincode.js:46:68) at info: [demo.js]: [Transaction Info] - Submitted: 0 Succ: 0 Fail:0 Unfinished:0 info: [bench-flow.js]: Executing command: docker-compose -f network/fabric-v1.4/2org1peergoleveldb/docker-compose.yaml down;docker rm $(docker ps -aq);docker rmi $(docker images dev* -q) Stopping peer0.org2.example.com ... done Stopping peer0.org1.example.com ... done Stopping orderer.example.com ... done Stopping ca.org2.example.com ... done Stopping ca.org1.example.com ... done Removing peer0.org2.example.com ... done Removing peer0.org1.example.com ... done Removing orderer.example.com ... done Removing ca.org2.example.com ... done Removing ca.org1.example.com ... done Removing network 2org1peergoleveldb_default f016f9dc219b Untagged: dev-peer0.org1.example.com-marbles-v1-a6cb10d46568463c6a5d95462ec8ed19eaa3145c0b63a2d905224fec13ad90ce:latest Deleted: sha256:abb1125a8d200187602d7369f0e75503c859a7c14bb97d318dada9757d012cfe Deleted: sha256:a3d43f00a19fc33c592682d0933484a4aa7f720f0aa18f81d85240cdc0967bd4 Deleted: sha256:9fe5ff84164e51834843853e9300eed6f8c08663916563c5303a3ccc1e35c00d Deleted: sha256:62c0eb59cb6e9deaf03e545eb0f6b8b4182d4c97dbe7accbd0a9a3a2d6942f82 info: [bench-flow.js]:

####################################### Test summary: 0 succeeded, 0 failed #######################################

info: [scripts/main.js]: Benchmark run successfully

aklenik commented 5 years ago

@nimaafraz the fabric adapter you use has been officially deprecated a few minutes ago. Try the Fabric-ccp adapter in the docs. Caliper can connect to any distributed fabric network, you just need to set the correct endpoints. If this is not the case, then probably it's a configuration issue on the fabric side

nimaafraz commented 5 years ago

Thanks, @aklenik. I finally managed to get it working and published a short guide for it here: https://medium.com/@nima.afraz/hyperledger-caliper-on-multiple-hosts-6bcd07492e07