search

eosnetworkfoundation / engineering

A workspace for documentation by Engineering primarily regarding process
MIT License
0 stars 0 forks source link

EVM Alerts for APAC Infrastructure #71

Closed wanderingbort closed 11 months ago

wanderingbort commented 11 months ago

Due to limitations in Amazon Web Services (AWS) offerings, it is not trivial to collect events from multiple AWS regions. Infrastructure alerts are flowing from the US region(s), but not APAC region(s).

There are better long-term solutions but, for now, it will be fastest and easiest to stand up parallel infrastructure in APAC alerting directly to IM. Automation used AWS EventBridge to send alerts from APAC to the US. This ticket is to do that.

See Also

engineering issue 68 - EVM Monitoring and Alerting - Phase 1

  1. eos-evm issue 602 - Funnel EVM Health Checks into CloudWatch
  2. engineering issue 48 - Collaborate with Operations on Unified Dashboarding Solution
  3. engineering issue 49 - Create Bot to Alert via IM on Specific Metrics
  4. eos-evm issue 603 - SMS Alerting for EVM Infrastructure Health Checks
  5. engineering issue 65 - Email Alerting for EVM Infrastructure Health Checks
  6. telegram-bot issue 1 - Open-Source This Repo
  7. engineering issue 57 - Create Telegram Service Account
  8. engineering issue 58 - Create EVM Testnet Alert Channel Using Telegram Service Account
  9. engineering issue 64 - Create EVM Mainnet Alert Channel Using Telegram Service Account
  10. engineering issue 66 - Fix EVM CloudWatch Alerts
  11. telegram-bot issue 2 - Human-Friendly Alerts
  12. engineering issue 71 - EVM Alerts for APAC Infrastructure
  13. telegram-bot issue 3 - Alert Bot Maintainer via Telegram on Errors
kj4ezj commented 11 months ago

I found that AWS EventBridge supports cross-account and cross-region event handling as of Sprint 2021. I am already using EventBridge to deliver CloudWatch alarm events to my notification system in the US, so I am working on pointing EventBridge in APAC to the existing US infrastructure. This will prevent the need for duplicate code/infrastructure across regions.

Relevant AWS Documentation

Here are some helpful resources showing this is possible and explaining how to do it.

kj4ezj commented 11 months ago

Done!

This was easy to implement and should be extremely reliable. I received an event from APAC on Telegram and via email for both the evm-mainnet and evm-testnet AWS accounts.