homebase-lite-backend: Regression in Homebase Lite Backend Causing Incorrect Endpoint Response

[benefacto]

Summary: A recent deployment caused a critical regression in the Homebase Lite Backend, leading to incorrect responses from the API. Specifically, the endpoint that was supposed to return DAOs JSON is instead returning Swagger HTML. This issue has caused significant disruption, and a rollback was performed to mitigate the immediate impact.

Details:

• Date/Time Reported: Today at 2:56 AM by Manank
• Initial Detection: Corey reported issues with Homebase functionality.
• Rollback: Manank rolled back to commit b5ecb55 at 3:31 AM as an immediate remedy.
• Issue Confirmation: Andrei confirmed the issue at 3:38 AM and provided additional details at 4:06 AM, stating the endpoint returns Swagger HTML instead of the expected JSON.
• Current Status: Post-rollback, the system is functional, but the issue remains unresolved in the current build.

Impact:

• Critical functionality for DAOs JSON retrieval impacted.
• Disruption caused inconvenience to users, notably Corey, who brought up the issue repeatedly.
• The issue occurred at a critical time, adding to its severity.

Potential Cause:

• Ben identified a possible route conflict issue and acknowledged the lack of post-deployment testing at 9:12 AM.

Next Steps:

• A thorough review of recent changes and additional post-deployment testing is recommended to identify and fix the root cause.
• Discuss the incident in the next team call to improve processes and avoid similar occurrences.
• Implementing enhanced test coverage to ensure robust functionality and prevent future regressions.

Update on Resolutions and Preventative Measures:

• We are expanding our testing suite to cover more scenarios and edge cases, especially focusing on endpoint functionalities and response types.
• Additional manual regression tests will be implemented to run post-deployment, verifying the correct behavior of all endpoints.
• These measures aim to strengthen our development process, enhancing the reliability and quality of our deployments.

Apologies and Acknowledgments:

• Ben acknowledged his mistake and offered to pause his work or step down if necessary.
• Andrei emphasized the importance of learning from the incident and moving forward as a team.

Action Required:

• Detailed investigation of the route conflict and incorrect API response.
• Implementation of additional safeguards in the deployment process, including enhanced test coverage, to prevent similar issues in the future.

[benefacto]

Pull request: https://github.com/dOrgTech/homebase-lite-backend/pull/21

[benefacto]

@ashutoshpw Above pull request is waiting on review, merge, deployment, post-deployment

[akdorg]

@Man-Jain is reviewing the PR as of now.

[akdorg]

This seems to be merged now.