search

hyperledger / caliper

A blockchain benchmark framework to measure performance of multiple blockchain solutions https://wiki.hyperledger.org/display/caliper
https://hyperledger.github.io/caliper/
Apache License 2.0
650 stars 403 forks source link

errors benchmarking with multiple orderers #392

Closed quangtdn closed 5 years ago

quangtdn commented 5 years ago

Hi everyone,

I recently have problems with running Caliper under fabric-ccp config with multiple orderers. I already successfully ran Caliper under fabric-ccp (connected to all 3 orderers) to benchmark a Kafka-based fabric network with 3 orderers.

For this time, I regenerate the configs file, bring up the same fabric network again. The fabric network still works fine. I can also successfully use Caliper to benchmark it under fabric.json config (i.e. connect to only 1 orderer). However, when I try it with fabric-ccp on 3 orderers, I receive the errors.

Update: I tried to run the basic default test of caliper 2org1peergoleveldb with fabric-ccp, but it gave me the same error. This is weird. Maybe Caliper might have clashed with my system in previous run !?

Updat1: Okay, I just rebuild the whole caliper (npm rebuild, then npm install). It works fine now. I think this is not a random error as I got the error on both my laptop and nodes on cluster.

info: [bench-flow.js]: ####### Caliper Test ####### info: [adapters/fabric-ccp]: Fabric SDK version: 1.1.0; TLS: none warn: [adapters/fabric-ccp]: Org1's registrar's materials found locally. Make sure it is the right one! warn: [adapters/fabric-ccp]: Org1's admin's materials found locally. Make sure it is the right one! warn: [adapters/fabric-ccp]: client0.org1.example.com's materials found locally. Make sure it is the right one! info: [adapters/fabric-ccp]: mychannel successfully created info: [adapters/fabric-ccp]: Sleeping 5s... info: [adapters/fabric-ccp]: Org1's peers successfully joined mychannel: peer0.org1.example.com,peer1.org1.example.com,peer2.org1.example.com info: [adapters/fabric-ccp]: Sleeping 5s... info: [adapters/fabric-ccp]: Installing chaincodes for mychannel... info: [packager/Node.js]: packaging Node from /users/quangtdn/caliper/src/contract/fabric/simple/node info: [adapters/fabric-ccp]: simple@v0 successfully installed on Org1's peers: peer0.org1.example.com,peer1.org1.example.com,peer2.org1.example.com info: [adapters/fabric-ccp]: Instantiating simple@v0 in mychannel. This might take some time... error: [adapters/fabric-ccp]: The following errors occured while instantiating simple@v0 in mychannel:

info: [scripts/main.js]: Benchmark run successfully

Thank you !

aklenik commented 5 years ago

@quangtdn Looks like the peer rejects your connection by force. Probably because of authentication issues. Double check that you use the same certs on both server and client/Caliper side. 

warn: [adapters/fabric-ccp]: Org1's registrar's materials found locally. Make sure it is the right one!
warn: [adapters/fabric-ccp]: Org1's admin's materials found locally. Make sure it is the right one!

There's probably some leftover certs in your Caliper machine. The network config specifies some user store paths, clean them up (the example end scripts do that).

quangtdn commented 5 years ago

@aklenik After rebuilding Caliper, I didn't find such errors again, so maybe the links to the certs in docker-compose and caliper's config file should be correct. In the unused script 'generate.sh' of Caliper, I note that there is a part that changes crypto file name [--some long number here--]_sk to 'key.pem'. I did reuse that for my own config generation later (so I don't have to edit my docker-compose for every time). I am not sure if it caused the errors.

aklenik commented 5 years ago

@quangtdn What I meant are the user stores created by the Fabric SDK. If a user name exists in the store, it will be loaded from there regardless of what cert path you specify. So a leftover cert from a previous experiment can cause problems. That's why loading a cert from the local store during the init phase is treated as a warning.

Is the issue solved by the way?

quangtdn commented 5 years ago

@aklenik Actually I still get the error sometimes. But I just rebuild Caliper anyway to solve it, as it does not take so much time.

aklenik commented 5 years ago

@quangtdn If you delete these dirs, does it solve the error without rebuilding Caliper?

rm -rf /tmp/hfc-*

quangtdn commented 5 years ago

@aklenik I am not so sure. I will try to do so next time I see the error.

quangtdn commented 5 years ago

@aklenik oh, it does help to solve the problem! Thank you xD!

niket-yende commented 5 years ago

Hi @quangtdn do you have a sample network config file for benchmarking kafka setup with multiple orderers?

I wanted to benchmark my use case which contains multiple orderers part of the kafka cluster.

Any help related to this would be really helpful. :)

quangtdn commented 5 years ago

Actually all you need to do is to add extra orderers into the fabric-ccp file. There is nothing to add for kafka brokers or zookeeper here. Caliper will do the benchmarking with just that.

So the real problem is to bring up a kafka-based fabric network by yourself, as default networks in caliper are all in solo mode. For my first simple kafka network, I just followed this one: https://medium.com/coinmonks/hyperledger-fabric-setup-with-multiple-peers-and-orderers-with-kafka-542023787a6d

niket-yende commented 5 years ago

I already have my kafka cluster with multiple orderers setup on an external VM.

I would like to know the following:

  1. How does caliper understand which orderer is associated with a peer? (Basically i have one peer0 part of org1 associated with orderer1 who is the leader, along with this there is one more orderer2 who will be associated with peer0 of org2)

Does it look-up the docker-compose configuration to figure out which orderer to send request to for a specific peer in a multi-orderer setup?

Note: My blokchain n/w is setup on another machine & caliper is assessing those endpoints through its n/w config file.

@quangtdn @aklenik

aklenik commented 5 years ago

@niket-yende Caliper doesn't know these details. The ordering service is a system-level service, so you can send endorsed transactions to any orderer node, they'll put it into the appropriate kafka topic.

Caliper does a round-robin load balancing between the peers and the orderers (based on either the transaction index, or the submitting client process). But if you need finer control, you can specify the target peers and orderer for each transaction individually. https://hyperledger.github.io/caliper/docs/Fabric_Ccp_Configuration.html#the-run-callback