search

Fantom-foundation / Norma

Integration project of Carmen and Tosca
GNU Lesser General Public License v3.0
2 stars 9 forks source link

[Bug] chain_execution_count, chain_id value extraction results in unexpected behavior. #245

Closed rpl-ffl closed 1 month ago

rpl-ffl commented 1 month ago

After upgrading to Sonic develop = 02b387, Norma has reported 2 newly-encountered unexpected behavior during testing:

  1. chain_execution_count appears to no longer be extractable. Norma tests if the sonic client's runtime behavior can be extracted or not using chain_execution_count:

https://github.com/Fantom-foundation/Norma/blob/4b529a0bc65473ee3ab881ddfe709119ae7c099e/driver/monitoring/node/prom_log_source_test.go#L89-L147

prom_log_source_test.go:145: Metric data not arrived within give time for: chain_execution_count

chain_execution and chain_execution_count seems to only appear in the testing section. (In other words, actual run looks for txpool_received, system_memory_used, etc.) Thus it could be that the string label changed or if we can check for other values instead of chain_execution to confirm value extraction.

Question

  1. chain_info starts to appear with the following value chain_info {} 1 where a float is expected in place of {}. 
monitoring: failed to parse log: cannot parse value from: {}; [chain_info {} 1]; strconv.ParseFloat: parsing "{}": invalid syntax

We can suppress this behavior on Norma front, but it would be nice to understand where the changes are coming from.

Question:

thaarok commented 1 month ago

There is a change of chain_execution/chain_execution_count introduced by go-ethereum dependency upgrade: The metric is visible in the /debug/metrics/prometheus output only when the samples count is not zero - it use ResettingTimer instead of Timer which was used in the older go-ethereum: https://github.com/Fantom-foundation/go-ethereum-sonic/blob/main/metrics/prometheus/collector.go#L125-L127 You need to handle missing chain_execution_count as a zero count.

I may try to fix the chain_info, however it seems it (or "chain_id") does not exist even in the main branch - I don't even see it in the Norma sources - how are you using it right now? Do you need it?

rpl-ffl commented 1 month ago

I may try to fix the chain_info, however it seems it (or "chain_id") does not exist even in the main branch - I don't even see it in the Norma sources - how are you using it right now? Do you need it?

Sorry, there was a typo before. It's always been chain_info, never chain_id. Norma does not use it. I have not seen it before and suppressing it by ignoring chain_info seems to result in a normal run.

thaarok commented 1 month ago

@rpl-ffl I have prepared fix here - please check attached example output if it matches your expectations: https://github.com/Fantom-foundation/Sonic/pull/219

rpl-ffl commented 1 month ago

@thaarok Thank you for the quick fix. Proceeding by ignoring chain_info for now.