search

hyperledger / caliper

A blockchain benchmark framework to measure performance of multiple blockchain solutions https://wiki.hyperledger.org/display/caliper
https://hyperledger.github.io/caliper/
Apache License 2.0
652 stars 404 forks source link

Caplier docker metrics produces strange values for Network IO #1165

Open davidkel opened 3 years ago

davidkel commented 3 years ago

Starts off ok

+---------------------------------------------------------------------------------------------------------+-----------+-----------+------------------+------------------+-----------------+------------------+-----------------+---------------+
| Name                                                                                                    | CPU%(max) | CPU%(avg) | Memory(max) [MB] | Memory(avg) [MB] | Traffic In [MB] | Traffic Out [MB] | Disc Write [MB] | Disc Read [B] |
|---------------------------------------------------------------------------------------------------------|-----------|-----------|------------------|------------------|-----------------|------------------|-----------------|---------------|
| dev-peer0.org1.example.com-fixed-asset-e9ce31fd539028d8475332bc2b4db419a749df5c954d9a91a8d30ab5c68e0426 | 24.98     | 21.63     | 19.4             | 19.3             | 135             | 58.9             | 0.00            | 0.00          |
|---------------------------------------------------------------------------------------------------------|-----------|-----------|------------------|------------------|-----------------|------------------|-----------------|---------------|
| dev-peer0.org2.example.com-fixed-asset-e9ce31fd539028d8475332bc2b4db419a749df5c954d9a91a8d30ab5c68e0426 | 24.53     | 20.93     | 19.4             | 19.3             | 134             | 57.8             | 0.00            | 0.00          |
|---------------------------------------------------------------------------------------------------------|-----------|-----------|------------------|------------------|-----------------|------------------|-----------------|---------------|
| cli                                                                                                     | 0.00      | 0.00      | 0.492            | 0.492            | 0.00            | 0.00             | 0.00            | 0.00          |
|---------------------------------------------------------------------------------------------------------|-----------|-----------|------------------|------------------|-----------------|------------------|-----------------|---------------|
| peer0.org1.example.com                                                                                  | 98.52     | 83.99     | 163              | 154              | 936             | 848              | 575             | 0.00          |
|---------------------------------------------------------------------------------------------------------|-----------|-----------|------------------|------------------|-----------------|------------------|-----------------|---------------|
| peer0.org2.example.com                                                                                  | 68.00     | 52.95     | 156              | 149              | 439             | 236              | 575             | 0.00          |
|---------------------------------------------------------------------------------------------------------|-----------|-----------|------------------|------------------|-----------------|------------------|-----------------|---------------|
| orderer.example.com                                                                                     | 30.33     | 21.56     | 68.6             | 68.4             | 302             | 586              | 709             | 0.00          |
+---------------------------------------------------------------------------------------------------------+-----------+-----------+------------------+------------------+-----------------+------------------+-----------------+---------------+

then as rounds continue

+---------------------------------------------------------------------------------------------------------+-----------+-----------+------------------+------------------+-----------------+------------------+-----------------+---------------+
| Name                                                                                                    | CPU%(max) | CPU%(avg) | Memory(max) [MB] | Memory(avg) [MB] | Traffic In [GB] | Traffic Out [GB] | Disc Write [GB] | Disc Read [B] |
|---------------------------------------------------------------------------------------------------------|-----------|-----------|------------------|------------------|-----------------|------------------|-----------------|---------------|
| dev-peer0.org1.example.com-fixed-asset-e9ce31fd539028d8475332bc2b4db419a749df5c954d9a91a8d30ab5c68e0426 | 33.86     | 28.60     | 19.5             | 19.5             | 0.562           | 0.273            | 0.00            | 0.00          |
|---------------------------------------------------------------------------------------------------------|-----------|-----------|------------------|------------------|-----------------|------------------|-----------------|---------------|
| dev-peer0.org2.example.com-fixed-asset-e9ce31fd539028d8475332bc2b4db419a749df5c954d9a91a8d30ab5c68e0426 | 32.54     | 27.87     | 19.5             | 19.5             | 0.561           | 0.272            | 0.00            | 0.00          |
|---------------------------------------------------------------------------------------------------------|-----------|-----------|------------------|------------------|-----------------|------------------|-----------------|---------------|
| cli                                                                                                     | 0.00      | 0.00      | 0.492            | 0.492            | 0.00            | 0.00             | 0.00            | 0.00          |
|---------------------------------------------------------------------------------------------------------|-----------|-----------|------------------|------------------|-----------------|------------------|-----------------|---------------|
| peer0.org1.example.com                                                                                  | 130.50    | 87.08     | 194              | 186              | -1.63           | -1.70            | 1.48            | 0.00          |
|---------------------------------------------------------------------------------------------------------|-----------|-----------|------------------|------------------|-----------------|------------------|-----------------|---------------|
| peer0.org2.example.com                                                                                  | 101.02    | 57.22     | 176              | 176              | 1.27            | 0.875            | 1.48            | 0.00          |
|---------------------------------------------------------------------------------------------------------|-----------|-----------|------------------|------------------|-----------------|------------------|-----------------|---------------|
| orderer.example.com                                                                                     | 31.95     | 22.80     | 74.8             | 72.8             | 0.708           | 1.40             | 1.51            | 0.00          |
+---------------------------------------------------------------------------------------------------------+-----------+-----------+------------------+------------------+-----------------+------------------+-----------------+---------------+

The above shows negative numbers (it wasn't the first report to move from MB to GB, a previous round did that but this could be relevant)

+---------------------------------------------------------------------------------------------------------+-----------+-----------+------------------+------------------+-----------------+------------------+-----------------+---------------+
| Name                                                                                                    | CPU%(max) | CPU%(avg) | Memory(max) [MB] | Memory(avg) [MB] | Traffic In [GB] | Traffic Out [GB] | Disc Write [GB] | Disc Read [B] |
|---------------------------------------------------------------------------------------------------------|-----------|-----------|------------------|------------------|-----------------|------------------|-----------------|---------------|
| dev-peer0.org1.example.com-fixed-asset-e9ce31fd539028d8475332bc2b4db419a749df5c954d9a91a8d30ab5c68e0426 | 52.74     | 39.80     | 35.7             | 33.6             | -0.0833         | -2.04            | 0.00            | 0.00          |
|---------------------------------------------------------------------------------------------------------|-----------|-----------|------------------|------------------|-----------------|------------------|-----------------|---------------|
| dev-peer0.org2.example.com-fixed-asset-e9ce31fd539028d8475332bc2b4db419a749df5c954d9a91a8d30ab5c68e0426 | 55.81     | 39.94     | 39.7             | 37.8             | -0.0835         | -2.04            | 0.00            | 0.00          |
|---------------------------------------------------------------------------------------------------------|-----------|-----------|------------------|------------------|-----------------|------------------|-----------------|---------------|
| cli                                                                                                     | 0.00      | 0.00      | 0.492            | 0.492            | 0.00            | 0.00             | 0.00            | 0.00          |
|---------------------------------------------------------------------------------------------------------|-----------|-----------|------------------|------------------|-----------------|------------------|-----------------|---------------|
| peer0.org1.example.com                                                                                  | 217.40    | 177.61    | 327              | 311              | 1.95            | 1.91             | 16.7            | 0.00          |
|---------------------------------------------------------------------------------------------------------|-----------|-----------|------------------|------------------|-----------------|------------------|-----------------|---------------|
| peer0.org2.example.com                                                                                  | 188.34    | 148.45    | 286              | 278              | -0.0770         | -2.10            | 16.7            | 0.00          |
|---------------------------------------------------------------------------------------------------------|-----------|-----------|------------------|------------------|-----------------|------------------|-----------------|---------------|
| orderer.example.com                                                                                     | 60.91     | 45.38     | 98.1             | 97.7             | 0.0145          | 0.00585          | 8.30            | 0.00          |
+---------------------------------------------------------------------------------------------------------+-----------+-----------+------------------+------------------+-----------------+------------------+-----------------+---------------+

I've also seen results with negative exponential values being shown

+---------------------------------------------------------------------------------------------------------+-----------+-----------+------------------+------------------+-----------------+------------------+-----------------+---------------+
| Name                                                                                                    | CPU%(max) | CPU%(avg) | Memory(max) [MB] | Memory(avg) [MB] | Traffic In [GB] | Traffic Out [MB] | Disc Write [GB] | Disc Read [B] |
|---------------------------------------------------------------------------------------------------------|-----------|-----------|------------------|------------------|-----------------|------------------|-----------------|---------------|
| dev-peer0.org1.example.com-fixed-asset-e9ce31fd539028d8475332bc2b4db419a749df5c954d9a91a8d30ab5c68e0426 | 36.88     | 30.05     | 23.6             | 22.7             | 1.41            | 713              | 0.00            | 0.00          |
|---------------------------------------------------------------------------------------------------------|-----------|-----------|------------------|------------------|-----------------|------------------|-----------------|---------------|
| dev-peer0.org2.example.com-fixed-asset-e9ce31fd539028d8475332bc2b4db419a749df5c954d9a91a8d30ab5c68e0426 | 35.20     | 29.91     | 22.2             | 22.2             | 1.41            | 712              | 0.00            | 0.00          |
|---------------------------------------------------------------------------------------------------------|-----------|-----------|------------------|------------------|-----------------|------------------|-----------------|---------------|
| cli                                                                                                     | 0.00      | 0.00      | 0.500            | 0.500            | 0.00            | 0.00             | 0.00            | 0.00          |
|---------------------------------------------------------------------------------------------------------|-----------|-----------|------------------|------------------|-----------------|------------------|-----------------|---------------|
| orderer.example.com                                                                                     | 36.18     | 26.48     | 90.2             | 86.5             | -2.47           | -995             | 3.12            | 0.00          |
|---------------------------------------------------------------------------------------------------------|-----------|-----------|------------------|------------------|-----------------|------------------|-----------------|---------------|
| peer0.org1.example.com                                                                                  | 170.93    | 90.58     | 252              | 244              | 2.93            | -1.88e+3         | 5.49            | 0.00          |
|---------------------------------------------------------------------------------------------------------|-----------|-----------|------------------|------------------|-----------------|------------------|-----------------|---------------|
| peer0.org2.example.com                                                                                  | 162.49    | 79.66     | 217              | 211              | 2.93            | -1.90e+3         | 5.49            | 0.00          |
+---------------------------------------------------------------------------------------------------------+-----------+-----------+------------------+------------------+-----------------+------------------+-----------------+---------------+

I assume the exp numbers are due to trying to match the values with the unit

davidkel commented 2 years ago

I'm unable to recreate this on my test system, but it is a VM rather than bare-metal which was where I think these came from. It's also interesting to note that there was some recording of Disc Write here whereas on my system I get nothing for disc read or disc write and docker stats shows the same. This might be due to fact that all volumes are docker-compose created volumes rather than the container file system based on this comment

Are you writing files to volumes? If so, docker doesn't show the block i/o to volumes, only to the copy on write filesystem in the container.

This would make the disk information unreliable and dependent on how the OS and docker interact