saumier
commented
3 months ago This issue happened again in May 2024 lasting over 24 hrs. As a result, a CloudFront distribution was placed in front of the LightSail instance. SSL and setup documented here https://github.com/culturecreates/culture-creates-wiki/wiki/How-to-setup-SSL-certificate-on-culturecreates.com-Wordpress
Incident Report
Summary
Corporate website was unavailable for about 3 days. Our monitoring tool, Datadog, checked every 30 minutes and reported that the Response Time exceeded 2 seconds (2000 ms) reaching around 0.5 minutes to load a page.
Timeline
Start: Saturday , Sep 02, 2023, 2AM ET End: Tuesday, Sep 06, 6AM ET
Lessons learnt
We learnt that LightSail has a limited 'CPU Burst Capacity', and when this is exceeded, the performance of the website drops to unacceptable (20 sec) response time.
In this case, the metrics on LightSail point to a sudden increase on the calls to the website, moving the CPU into the burstable zone on September 2nd, and then leading to a drain of all the CPU burst capacity. The burst capacity is meant to be short. The sustained CPU burst shown (see graph below) used up all the accumulated Burst capacity.
It is not clear why the traffic to the website suddenly increased, lasted for 3 days, and then went away. Nothing was changed on our side. A guess is a random bot indexing or attack or automatic software update causing the external network traffic to rise.
Here are the metrics of the incoming network traffic that corresponds to the dates of the down time.
Action items
Caitlin and Tammy are planning to redo the corporate website and have mandated our graphic artist/UX designer to propose ideas. We should wait for the plan for the new website before working on the hosting improvements.