[Telemetry] Mainnet Monitoring - Rebase Health Module

[jessicadaugherty]

Objective

Following the 0.9.1.1 Chain Halt Post-Mortem, there are action items re: monitoring mainnet (and ideally testnet) that we need to get better coverage of to increase our chances of catching bugs/errors before releases are in production, as well as helping us triage issues in the event of a release/production crisis.

Due to the need to process state dumps and aggregate by network actors, we need a dedicated exporter available rather than making enhancements to an existing exporter.

Origin Document

A health module was designed and built but remained inactive that includes:

• Consensus Metrics: TendermintBFT block by block metrics
• State Metrics: Sessions, relays, jailing, etc.
• Transaction Metrics: Sender, processing time, total valid/invalid txs, etc.
• Data Size Metrics: State and block DBs
• Lifecycle Metrics: ABCI state machine processing performance

Consensus, State and Transaction metrics are the most relevant when triaging crises like a chain halt, while data size and lifecycle metrics help us observe the performance of state size and transitions through the state.

Goals

• [ ] Rebase the health module exporter based on the current version of pocket-core

Deliverable

• [ ] Review the existing health module and address merge conflicts
• [ ] Understand the Go structures being being exported
• [ ] Retrieve the data as JSON (similar to tendermint endpoints)
• [ ] Design the promethesus exporter (TBD)

Non-goals / Non-deliverables

• Setting up monitoring alerts
• Visualizing exporter data

General issue deliverables

• [ ] Update the appropriate CHANGELOG
• [ ] Update any relevant READMEs (local and/or global)
• [ ] Update any relevant global documentation & references
• [ ] If applicable, update the source code tree explanation
• [ ] If applicable, add or update a state, sequence or flowchart diagram using mermaid

Testing Methodology

---

Creator: @jessicadaugherty

[jessicadaugherty]

Questions

1. If we need min. 2 state dumps for diffs, how do we handle that with this exporter, or do we always need data from another node runner?
2. At what frequency is this queried? On an as needed or ongoing basis?

[Olshansk]

@Gustavobelfort Making a "public note to self" for us to sync on this offline.

1. Look at the branch related to this PR: https://github.com/pokt-network/pocket/issues/360
2. See the impact / feasibility of rebasing this on top of master
3. Understand the Go structures being being exported
4. Retrieve the data as JSON (similar to tendermint endpoints)
5. Design the promethesus exporter (TBD)

[jessicadaugherty]

Refactored Deliverables

• Confirm flag exists that enables/disables the health module in the codebase
• If flag exists, document risks related to activating the health module and the data field inconsistencies and merge

@Gustavobelfort @Olshansk

[Gustavobelfort]

Currently the state of the health module is as follows:

Problems encountered in the codebase

• StateSize

  • Broken delta logic makes returns be negative of zeroed
  • We should return either: a delta + an absolute value for the state size or just an absolute value for it

• Block Size

  • Inconsistent value compared to POKT Scan, we should double check with them in order to validate how the block size is calculate

• API should be able to allow either paginating or opting in/out of the data contained in the blocks (claims, proofs , transactions).

  • Currently all the data is returned at once and that might be problematic

• Not clear wether the returned information is "partial" or not (ie: historic vs live query)

• Lack of clarity on null returns

  • Consensus Metrics is null
    • Needs to be validated if actually is working by generating a local chain halt
  • The API does not indicate that the data has been pruned and communicates wrongly the results as a nil object
  • Is information null because block is too old? is information null because block is too new?
  • We should be able to communicate between the different scenarios (data missing, data incomplete, data complete)

Working properly

• Total relays

• Total transactions

• Session Metrics

  • Session Generation Times
  • Sessions Generated

• Lifecycle Metrics

  • Apply Block Time
  • Begin Block
  • Deliver Txs
  • End Block

Conclusion

After discussing with @iajrz we don't think that the module is ready to be merged in the codebase, some unwanted side effects in pocket-core might pop up if we decide to do so.

Ideally we should review the metric requirements of the health module in order to better design what should be returned, utilize the bits and pieces of the code that are working and either trim out the pieces that do not work or fix them basing ourselves on the design doc created beforehand, only then decide about merging.

[Olshansk]

Closing this out as the work is no longer relevant. We can reference this PR if we ever choose to pick it up again.