hyperledger / caliper

A blockchain benchmark framework to measure performance of multiple blockchain solutions https://wiki.hyperledger.org/display/caliper
https://hyperledger.github.io/caliper/
Apache License 2.0
650 stars 402 forks source link

Determine the true performance of a fabric network when using peer gateway service #1443

Open davidkel opened 2 years ago

davidkel commented 2 years ago

In a fabric network it would be normal for organisations to be submitting transactions concurrently. When using the legacy SDKs the would not be as much bias to the org that was running submission tests so it would be indicative of the overall performance of the network when running from any of the organisations (however it can be observed that the legacy sdks will still likely not provide an even distribution for endorsement against peers with the same block height). The only differentiating factor comes down to how fast a peer processes a committed block in order to indicate a completed submission and that can vary from org to org depending on the peer resources (eg cpu, speed of disk etc), but that will also affect how fast an endorsement is done on that peer anyway which influences how fast a submit will take anyway so still relates to the overall network performance.

With the introduction of the fabric-gateway connector, this dynamic changes. There is now bias towards the org running the test as the gateway peer does an initial endorsement and also does the work of processing the responses as well as submitting to the orderer. This means now that the resources allocated to that peer affect the performance seen by a single organisation and will differ if another org's gateway peer is allocated different resources, so the results are now more specific to a single org rather than before where it wasn't quite so important as a caliper run from 1 org won't look any different to a caliper run from another org when submitting transactions alt(evaluate tests are of course org specific as well, but evaluate transactions should not use peers from other organisations if at least due to trust issues)

To be able to determine the theoretical maximum performance of submit requests we should be able to load balance requests across gateway peers in both the same organisation and different organisations. Being limited to 1 organisation in this case will reduce the max possible theoretical TPS because the 1 org does more work than other orgs.

davidkel commented 1 year ago

Load balancing in the fabric-gateway connector is covered here #1210 In a multi-org network could we have a workload that submits the same thing to all the orgs ? This might do it but we would need org targeting which is #1272