Closed dev-aravind closed 1 year ago
@troughc Please review this incident report. You can close it when you are done.
@saumier I like the modification you made to the report.
@troughc Please read the report and close.
Read and closed. Thank you for this.
Incident Report
Summary
Unable to open the detailed view of most of the events irrespective of the calendar. The issue lasted less than a day. The root cause of the issue was a technical glitch.
Timeline
2023-JUN-14
About 10 hrs between problem was detected and fixed.
[16 h 20]
Email from Signé Laval reporting LIVE failures. Laurent Blais-Sénéchal Chargé de communication - Signé Laval
[20 h 01]
Gregory emails Laurent to acknowledge the issue and say we are working on it.
[20 h 10]
Gregory alerts @everyone in Slack channel. https://culturecreates.slack.com/archives/C02B18SN3FU/p1686787813876419
Gregory starts working to fix problem.
Gregory monitor the backend log in the log management system, datadog, and identified the root cause and identified that the issue is happening while formatting an organization/person linked to an event.
Gregory opened an issue #545 and assigned it to Suhail.
[21 h 42]
Gregory sends email to Laurent at Signé Laval that we have identified the problem and are fixing it.
Suhail analyzed the log and took a deep dive into the issue and found that the issue is related to formatting organization or person with no dynamic fields assigned to it.
Did a hotfix and released a new version of the backend.
[1 h 51]
Suhail releases a fix in production and sends email
What went well
List of things that went well. For example,
What went wrong