Add monitoring for secondary EVM events forwarded by validators

[daniel1302]

Background

We determine how many Ethereum events have been forwarded by each validator in the last 10 days. If it is low we fire the alarm.

Issue

At the moment We do not split it by chain. We count all events from all the EVM chains. But We are going to add Arbitrum soon and We would like to see when validators have problems with their Ethereum and Arbitrum separately

TODOs:

• [ ] Update vega monitoring to determine what chain events come from by each validator
• [ ] Update dashboard: https://monitoring.vega.community/d/b0816c52-d2f1-4d4d-911b-2b162aef470c/vega-mainnet-network?orgId=1
• [ ] Update alarm (https://monitoring.vega.community/alerting/grafana/f90f2d0e-593c-40f1-8a63-94fff7512b18/view)
• [ ] Test on m3.vega.community
• [ ] Get vega-monitoring changes merged
• [ ] Release new version
• [ ] Bump version in ansible
• [ ] Release to other servers

[ValentinTrinque]

SQL request for the Grafana chart.

WITH chain_event_count AS (
  SELECT submitter, COUNT(*) AS count, last(vega_time, vega_time) AS last, attributes->>'source-chain-id' as chain_id
  FROM comet_txs
  WHERE
      command = 'Chain Event' AND attributes ? 'source-chain-id'
    AND
      vega_time >= NOW() - INTERVAL '10 days'
  GROUP BY submitter, attributes->>'source-chain-id'
)
SELECT
  COALESCE(c.count, 0),
  CONCAT(n.name, ' (', c.chain_id , ') ', ' [', COALESCE(TO_CHAR(c.last,'DD Mon HH24:MI'), '-'), ']') AS description
FROM validator_nodes n
LEFT JOIN chain_event_count c ON c.submitter = encode(n.vega_pub_key, 'hex')
ORDER BY description