Self-monitor validation performance

[carlhua]

If rippled is configured as a validator, it should monitor its own performance. Currently, we do log cases where build one ledger and later accept a different one as fully validated. But it would really be nice to have statistics on validation performance.

Specifically, we should track: 1) The fraction / indexes of ledgers we are not sending validations for 2) The fraction / indexes of ledgers we send validations for, but they don't match the ledger hash we later accept as fully valid 3) The fraction / indexes of ledgers we send validations for and which are subsequently accepted as fully valid (happy path)

We can, and should, track this for non validators too. Instead of being based on validations sent, it can be based on ledgers built as a result of consensus rounds.

Exported from RIPD-1287

[intelliot]

Writing this info to a log file seems best, but we don't want to create another log. We should be able to estimate the file's rate of growth. Could offer sample scripts for how to view/aggregate the data.

May be something for @thejohnfreeman / @Bronek to look into.

[Bronek]

I am starting to think that writing this to the existing perflog would be a decent solution:

• the data gets persisted between restarts etc.
• we get to use the existing perflog infrastructure
• perflog would still have to be filtered e.g. by Grafana scripts to find the relevant info, but I guess it already is anyway

At the same time, we could also add some of this information to get_counts