Closed saumier closed 10 months ago
@saumier I've updated the incident report according to my understanding of what happened on that particular day. Please review it and do give suggestions if you want to update this.
@dev-aravind Thanks for writing up this incident report. Your timeline was perfect. I added the "image timeline" only because it was communicated to the client.
I have 2 suggestions to add:
@saumier I've made changes to the report. Please review it and let me know about what you think about my process improvement ideas.
@dev-aravind Good improvement. Thx. Lets discuss at the next standup how we can have all our team receive the emails from Datadog. We have a list called support@culturecreates.com that includes all our development team. The emails for P1 could be sent to that list instead of admin@culturecreats.com. Once this is setup, I will close this issue.
We decided to use support@culturecreates.com for all Datadog monitors and synthetic tests.
Incident Report
Summary
Customer websites had no events because of the Footlight API server https://api.footlight.io/api/ giving 502 Bad Gateway. The issue lasted less than a day. The root cause of the issue was related to the deployment script, as the port number for the Open-Api service in production was not updated to 3030 from 80 during the release of version 1.11.4.
Timeline
2023-09-29 04:36
The Open-API service went down after the release of v1.11.4
[07:47]
@saumier alerts @everyone in the Slack channel about the Open-API being down and no events on the customer websites.
[07:48]
@sahalali starts working on the issue.
[08:25]
@sahalali updated the migration script to fix the issue and did a hotfix and released a new version of the Open API
[08:30]
@saumier marked the incident as resolved.
The following timeline was shared with the client. It was generated with Datadog (our server logging and monitoring software) but the "declared" timestamp is later than when the incident was declared on Slack. Slack was the primary communication tool.
What went well
What went wrong
Where we got lucky
Action items
Process improvements
To ensure a seamless release process , we should create a dedicated issue and break down the process into subtasks. These subtasks should address tasks like verifying the staging environment, merging with production, and thoroughly testing client websites and related footlight services after the release.
To ensure faster recognition in the event of a server outage, I would like Datadog to alert at least one team member from each of the regions (Canadian and Indian) since we have different work hours due to the time zone differences. This way, both teams are notified and can take immediate action in case any of our service goes down.