search

hyperledger / caliper

A blockchain benchmark framework to measure performance of multiple blockchain solutions https://wiki.hyperledger.org/display/caliper
https://hyperledger.github.io/caliper/
Apache License 2.0
650 stars 402 forks source link

Different definition of throughput on the Caliper FAQ than the actual code. #1418

Open stavrosdim opened 2 years ago

stavrosdim commented 2 years ago

Which Caliper version are you using?

v0.5.0

Which Node.JS version are you using?

v10.24.1

Which operating system are you using?

Ubuntu 22.04 LTS

Please provide some context for your error. For example, when did the error occur? What were you trying to achieve, and how?

I was using caliper to benchmark my Hyperledger Fabric network, and i was highly interested in the Throughput (TPS) value. Apparently at the https://hyperledger.github.io/caliper/v0.5.0/caliper-faq/ it is insisted that: "The throughput is calculated by Succ/(last committing time - first submitting time), here only successful committed Txs will be calculated.", however when i run an experiment with zero successful TXs i was still getting a valid Throughput which was odd. So i researched the code and at the file report.js i found out that the Throughput (TPS) value is calculated as (Succ+Fail)/(last committing time - first submitting time). Which is the intended way to caclulate the Throughput ?

What was the observed incorrect behavior?

Positive throughput value with zero successful TXs.

Please provide the error logs and their surroundings.

Name    Succ    Fail    Send Rate (TPS) Max Latency (s) Min Latency (s) Avg Latency (s) Throughput (TPS)
CreateAsset 0   5001    50.0    -   -   -   32.6

Please provide your benchmark configuration file content, if possible.

No response

Please provide your network configuration file content, if possible.

No response

Please provide your workload module content, if possible.

No response

Please provide any additional information you deem relevant to the error.

No response

davidkel commented 2 years ago

I'm guessing the FAQ has never been updated. The code has been that way for a while by the looks of things

stavrosdim commented 2 years ago

I see. Thanks !

davidkel commented 2 years ago

@stevedimaras I think this is a documentation bug, many thanks for reporting

davidkel commented 1 year ago

re-opening as we still need to fix the documentation here, so will use this issue to track that