search

hyperledger / fabric

Hyperledger Fabric is an enterprise-grade permissioned distributed ledger framework for developing solutions and applications. Its modular and versatile design satisfies a broad range of industry use cases. It offers a unique approach to consensus that enables performance at scale while preserving privacy.
https://wiki.hyperledger.org/display/fabric
Apache License 2.0
15.65k stars 8.81k forks source link

Peer metrics ledger_transaction_count exported by Prometheus is different from the transaction count in the chaincode #4949

Open stkaspar opened 1 month ago

stkaspar commented 1 month ago

Description

I checked the transaction count via the chaincode or SDK should be like 500, but in the Grafana(from Prometheus), there was only like 350(ledger_transaction_count), less than the number from the chaincode.

And I know there were some transactions with the status MVCC_READ_CONFLICT, but I didn't see them in the Grafana. If you fetch the transactions via chaincode or SDK, it's expected you won't get transactions with the above status. So it's expected the number of transaction count in the Grafana should be greater than the number from the chaincode.

Here are the peer metrics I get from the endpoint /metrics, you can see that the ledger_transaction_count only shows the transactions with validation_code=VALID, it didn't count another transaction status like MVCC_READ_CONFLICT

...
# HELP ledger_transaction_count Number of transactions processed.
# TYPE ledger_transaction_count counter
ledger_transaction_count{chaincode="_lifecycle:syscc",channel="mychannel",transaction_type="ENDORSER_TRANSACTION",validation_code="VALID"} 3
ledger_transaction_count{chaincode="basic:1.0.1",channel="mychannel",transaction_type="ENDORSER_TRANSACTION",validation_code="VALID"} 226
ledger_transaction_count{chaincode="unknown",channel="mychannel",transaction_type="CONFIG",validation_code="VALID"} 3
...

FYI.
Official Metrics doc
https://hyperledger-fabric.readthedocs.io/en/release-2.2/metrics_reference.html

Steps to reproduce

  1. Start the Fabric's services peer, orderer, prothemetus and grafana
  2. Deploy the smart contract
  3. Invoke the chaincode (Could try to invoke the chaincode multiple times at the same time to reproduce transaction status MVCC_READ_CONFLICT)
  4. Check the transaction count in the Grafana and via chaincode or SDK
denyeart commented 1 month ago

ledger_transaction_count has a validation flag label associated with it. If you have two transactions, where one is valid, and one is invalid with MVCC_READ_CONFLICT, you will get two counters incremented, e.g.:

ledger_transaction_count{chaincode="basic:1.0.1",channel="mychannel",transaction_type="ENDORSER_TRANSACTION",validation_code="VALID"} 1
ledger_transaction_count{chaincode="basic:1.0.1",channel="mychannel",transaction_type="ENDORSER_TRANSACTION",validation_code="MVCC_READ_CONFLICT"} 1

This is working as intended.

stkaspar commented 4 weeks ago

ledger_transaction_count has a validation flag label associated with it. If you have two transactions, where one is valid, and one is invalid with MVCC_READ_CONFLICT, you will get two counters incremented, e.g.:

ledger_transaction_count{chaincode="basic:1.0.1",channel="mychannel",transaction_type="ENDORSER_TRANSACTION",validation_code="VALID"} 1
ledger_transaction_count{chaincode="basic:1.0.1",channel="mychannel",transaction_type="ENDORSER_TRANSACTION",validation_code="MVCC_READ_CONFLICT"} 1

This is working as intended.

Thanks for your reply.

In my environment, I didn't see the below output, that's quite strange

ledger_transaction_count{chaincode="basic:1.0.1",channel="mychannel",transaction_type="ENDORSER_TRANSACTION",validation_code="MVCC_READ_CONFLICT"} 1

Or does it have any filter parameters when calling the /metrics endpoint?

denyeart commented 4 weeks ago

There is no filter for /metrics endpoint. I tried calling the /metrics endpoint with one valid and one invalid transaction and I get back:

ledger_transaction_count{chaincode="basic:1",channel="mychannel",transaction_type="ENDORSER_TRANSACTION",validation_code="MVCC_READ_CONFLICT"} 1
ledger_transaction_count{chaincode="basic:1",channel="mychannel",transaction_type="ENDORSER_TRANSACTION",validation_code="VALID"} 1
stkaspar commented 3 weeks ago

Then anything I can do for the result from the /metrics? cuz now the metrics are not correct and can't be a health index for my application

stkaspar commented 3 weeks ago

There is no filter for /metrics endpoint. I tried calling the /metrics endpoint with one valid and one invalid transaction and I get back:

ledger_transaction_count{chaincode="basic:1",channel="mychannel",transaction_type="ENDORSER_TRANSACTION",validation_code="MVCC_READ_CONFLICT"} 1
ledger_transaction_count{chaincode="basic:1",channel="mychannel",transaction_type="ENDORSER_TRANSACTION",validation_code="VALID"} 1

Can I confirm that each transaction, even the transaction's validation code is VALID or MVCC_READ_CONFLICT or other status, it should show from the /metrics, right?

denyeart commented 3 weeks ago

There is no filter for /metrics endpoint. I tried calling the /metrics endpoint with one valid and one invalid transaction and I get back:

ledger_transaction_count{chaincode="basic:1",channel="mychannel",transaction_type="ENDORSER_TRANSACTION",validation_code="MVCC_READ_CONFLICT"} 1
ledger_transaction_count{chaincode="basic:1",channel="mychannel",transaction_type="ENDORSER_TRANSACTION",validation_code="VALID"} 1

Can I confirm that each transaction, even the transaction's validation code is VALID or MVCC_READ_CONFLICT or other status, it should show from the /metrics, right?

I checked the code... for every transaction in a block, regardless of the validation status, will result in a counter increment for ledger_transaction_count with corresponding validation_code label.

stkaspar commented 3 weeks ago

There is no filter for /metrics endpoint. I tried calling the /metrics endpoint with one valid and one invalid transaction and I get back:

ledger_transaction_count{chaincode="basic:1",channel="mychannel",transaction_type="ENDORSER_TRANSACTION",validation_code="MVCC_READ_CONFLICT"} 1
ledger_transaction_count{chaincode="basic:1",channel="mychannel",transaction_type="ENDORSER_TRANSACTION",validation_code="VALID"} 1

Can I confirm that each transaction, even the transaction's validation code is VALID or MVCC_READ_CONFLICT or other status, it should show from the /metrics, right?

I checked the code... for every transaction in a block, regardless of the validation status, will result in a counter increment for ledger_transaction_count with corresponding validation_code label.

Thanks for the code checking and could you share where the code is?

denyeart commented 3 weeks ago

There is no filter for /metrics endpoint. I tried calling the /metrics endpoint with one valid and one invalid transaction and I get back:

ledger_transaction_count{chaincode="basic:1",channel="mychannel",transaction_type="ENDORSER_TRANSACTION",validation_code="MVCC_READ_CONFLICT"} 1
ledger_transaction_count{chaincode="basic:1",channel="mychannel",transaction_type="ENDORSER_TRANSACTION",validation_code="VALID"} 1

Can I confirm that each transaction, even the transaction's validation code is VALID or MVCC_READ_CONFLICT or other status, it should show from the /metrics, right?

I checked the code... for every transaction in a block, regardless of the validation status, will result in a counter increment for ledger_transaction_count with corresponding validation_code label.

Thanks for the code checking and could you share where the code is?

The stats update happens as soon as the block is committed: https://github.com/hyperledger/fabric/blob/main/core/ledger/kvledger/kv_ledger.go#L726

Here's the ultimate counter increment: https://github.com/hyperledger/fabric/blob/main/core/ledger/kvledger/metrics.go#L69-L74