[Implement KPIs] Implement API Uptime monitoring

[SamantaCasal]

Previous spike:

1. Develop cron job that queries the API. Problem, it would run on our infra
2. Look into a third party tool such as https://grafana.com/products/cloud/features/#synthetic-monitoring
3. Currently we have metrics from the cluster PoV (here ), but not from the client. The KPI aims to add visibility from the outside user’s PoV.
4. Idea: new CRON job that queries the API (decide later which ones to hit) and records whether it was successful or not, and the reason. Use retries.

TODO:

• Look into which tools are available for this and whether it should run entirely outside our infrastructure. Maybe ask infra.
• Look into https://grafana.com/products/cloud/features/#synthetic-monitoring

Source: DOC

[sebastian-xlabs]

I have spoken with @M-Picco and the endpoints we will use to query the http_code are:

https://api.wormholescan.io/api/v1/health
https://api.wormholescan.io/api/v1/operations
https://api.wormholescan.io/api/v1/x-chain-activity/?timeSpan=7d

We can use the AWS Operations account to query the application using a lambda function that will be triggered by an EventBridge rule (cron) every minute. The metrics can be stored in CloudWatch and we can get them using Grafana and setting CloudWatch as a data source.

[SamantaCasal]

Update:

https://grafana.ops.xlabs.xyz/d/adhnkd4xndddsa/wormscan-metrics?orgId=1

[sebastian-xlabs]

The PR is this https://github.com/XLabs/ops-aws-iac/pull/342