Beef up alerting based on recent outage

shift-org / shift-docs

Shift2Bikes: website and calendar for shift and pedalpalooza

https://shift2bikes.org

Other

22 stars 17 forks source link

Beef up alerting based on recent outage #431

Open onewheelskyward opened 2 years ago

onewheelskyward commented 2 years ago

Looks like we had a cpu usage spike followed by a significant network outage this morning. Add alerts to track the following:

CPU usage std dev alert
Network usage dropping to 0 or near zero
average response time from nginx, by api endpoint and static image?
Anything else the full analysis finds.

fool commented 1 month ago

@onewheelskyward could you write a quick doc about how to see the alerting you've set up? I get lost in AWS dashboard every time I look at it...

I'm also hopeful that we can somehow pipe the uptime monitor into slack via a webhook, if AWS can handle that? I tried a few free uptime monitor services and none of them seemed to work via webhook on the free plan :|

onewheelskyward commented 1 month ago

I've got a datadog synthetic test running, I should change the email to go to bikecal. We can definitely wire up a slack webhook for uptime.

fool commented 1 month ago

that'd be super swell, thank you!!

On Tue, Oct 15, 2024 at 8:20 AM Andrew Kreps @.***> wrote:

I've got a datadog synthetic test running, I should change the email to go to bikecal. We can definitely wire up a slack webhook for uptime.

— Reply to this email directly, view it on GitHub https://github.com/shift-org/shift-docs/issues/431#issuecomment-2414272951, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABX5IBVQ376S2CKQDUOQIDZ3UXDZAVCNFSM6AAAAABP6C4AMKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJUGI3TEOJVGE . You are receiving this because you commented.Message ID: @.***>