search

harmony-one / harmony

The core protocol of harmony
https://harmony.one
GNU Lesser General Public License v3.0
1.46k stars 286 forks source link

Improve synthetic monitoring and alerting of RPC methods #3789

Open givp opened 3 years ago

givp commented 3 years ago

Using AWS synthetic canary tests, we can simulate typical user experiences as it relates to RPC request and response flows. This will not only allow us to monitor response times but will also allow us to perform regression testing by ensuring response schemas are in tact and accurate returned values are present. Refer to this issue for a list of methods to be tested.

  1. Go to AWS > Cloudwatch > Synthetics > Canaries
  2. Create a new Canary and select "API Canary" blueprint Image 2021-06-18 at 4 32 04 PM png
  3. Add a new HTTP request Image 2021-06-18 at 4 35 19 PM png
  4. Add request URL, headers, and request body. Make sure to select "Capture headers and response body" Image 2021-06-18 at 4 37 37 PM png
  5. Using the script editor, use Javascript to inspect the response. For example, ensure a 2xx status code is returned and inspect the response payload to ensure certain fields are present and they match expected results. Image 2021-06-18 at 4 40 31 PM png
  6. Specify the frequency of your test Image 2021-06-18 at 4 44 05 PM png
  7. Finally, enable alarms for when success rates or endpoint response times fall below/above an acceptable threshold Image 2021-06-18 at 5 14 16 PM png

You can now use the alarm like any other Cloudwatch alarm. It can trigger an email, send a message to PagerDuty and update https://status.harmony.one

LeoHChen commented 3 years ago

I tried it once to create a canary to test one RPC API. This is super nice.

LeoHChen commented 3 years ago

We can open a bounty to use either AWS CLI or CloudFormation to build a complete RPC monitoring hub using canary automation. The list of RPC calls can be found in https://api.hmny.io