Port the metrics db out of Explorer infrastructure and into N1

[Trisfald]

After an initial investigation I propose here the following plan in order to port the telemetry infrastructure into N1.

First milestone - feature parity:

• [x] Setup a Postgres database in N1
  • [x] Understand and clean up the schema used in the explorer database (ref)
  • [x] Understand what data is needed and what is not (data sent by nodes)
  • [x] Understand the telemetry protocol details (format of the data, frequency, load, how is it persisted and queried)
  • [x] Create the DB instance in GCP (with high availability). One instance for both mainnet and testnet, the database will be different though
• [x] Create the middle layer service to receive telemetry data
  • [x] Handling schema creation
  • [x] Expose an HTTP endpoint accepting POST requests with JSON payload
  • [x] Write the data received into the database
  • [x] Metrics and health checks
  • [x] Packaging and CI
  • [x] Deploy multiple replicas for high availability
• [x] Setup alerting and monitoring for the new infrastructure
• [x] Configure the new Postgres datasource in N1 Grafana (docs)
• [x] Make a copy of the dashboard(s) and use the new datasource from NearOne

Second milestone - migration:

• [x] Make sure the new solution works and is stable
• ~[ ] Ask Pagoda to redirect telemetry traffic to the new service~
• [x] Update dashboards
• [x] Add the new telemetry endpoint to the node's config
• [ ] (next release) Deprecate and remote the old telemetry endpoint

Third milestone - nice to have:

• [x] Autodetect the chain ID
  • [x] TBD if we want to do this
  • [x] Modify the telemetry protocol to include the chain ID
  • [x] Switch to a single endpoint and use chain ID to distinguish between mainnet and testnet
• [ ] Remove Pagoda telemetry forwarding
  • [x] Update the node config to use the new telemetry endpoint. A single one if chain detection has been implemented, otherwise stick to mainnet and testnet.
  • [ ] Later ask Pagoda to remove the HTTP forwarding, if no new requests hit the old endpoints.

[Trisfald]

Update as of Fri May 10:

• Created a page in outline for this project: link.
• Fleshed out the implementation plan and the design.
• Progress made on the data model: understanding how telemetry data is stored and queries, which data is useful and which isn't, clearing up unimplemented and legacy fields.
• Started working on the thin HTTP service layer.

[Trisfald]

Fri 17 May Update:

• Provisioned the database on GCP
• Completed the work on the HTTP server
• Deployed the necessaries resources on GCP with monitoring, etc
• Done synthetic load test and functional test with a local validator

[Trisfald]

Fri May 24 update:

• Setup of the new telemetry subdomain: https://telemetry.nearone.org
• Test with testnet canaries
• Added the new datasources to grafana
• Created a dashboard with the new datasources

Some changes to the overall plan:

• Pagoda will simply redirect telemetry traffic to N1 (probably next week)

[Trisfald]

Redirecting telemetry traffic from Pagoda to N1 is non trivial, so I'm falling back to have both endpoints for a while and rotating out the Pagoda endpoint in the future.

[Trisfald]

The new telemetry endpoint is active and nodes are sending traffic to it. Also, automatic chain detection has been implemented.