As a DAWSON System Administrator, so that I can quickly triage issues in production, I need to be alerted when Lambdas produce errors.
Pre-Conditions
Acceptance Criteria
Errors continue to be logged to Cloudwatch logs via the Application logger.
Errors and exceptions are also reported to a service that helps us triage and address them, giving us an easy way to connect them to github issues and the deployments they were released.
Lambdas that need to be included:
streams_<ENV>_<COLOR>
api_<ENV>_<COLOR>
api_async_<ENV>_<COLOR>
public_api__<ENV>_<COLOR>
cognito_post_confirmation_lambda_<ENV>_<COLOR>
cognito_post_authentication_lambda_<ENV>_<COLOR>
update_petitioner_cases_lambda_<ENV>_<COLOR>
send_emails_<ENV>_<COLOR>
check_case_cron_<ENV>_<COLOR>
pdf_generator_<ENV>_<COLOR>
seal_in_lower_<ENV>_<COLOR>
cognito_authorizer_<ENV>_<COLOR>
migration_segments_lambda_<ENV>_<COLOR>
migration_lambda_<ENV>_<COLOR>
Additionally, uncaught client side errors should also be reported to a service to help us triage them.
This service provides configurable alert rules allowing DAWSON System Administrators the ability to be manage how they are alerted when errors occur. NOTE: It may make sense to have the client side error handling be its own vertical slice.
Notes
Should help prevent silent errors like the ones discovered in Bug 10072
Ref: #10043
Perhaps a spike is warranted to determine the best error error handling and monitoring service. The court's current preference is Sentry, and appreciate that there has already been experience on the team with Honeybadger.
Ways to make a Lambda fail
breaking the code so that it throws an error
make an entity fail validation by updating the record in the database
Ways to make Client side errors
breaking the code so a JavaScript error is thrown
Tasks
Test Cases
Lambda throws an Error or logs an error, we would expect the service to receive and store as metadata about the error:
a stack trace of the error, if one is available
any included context that is passed along via applicationContext.logger.error('message', { contextKey: contextValue })
the lambda that triggered the error
the requestId of the lambda that triggered the error
any requests that involve a password or tokens do not have that information included
Errors and exceptions are still logged to the Cloudwatch logs for debugging and troubleshooting
The following errors are not sent to the service:
NotFoundError
UnkonwnUserErrror
UnauthorizedError
UnprocessableEntityError
ServiceUnavailableError
The service sends email alerts when new errors occur. The information in the emails depends on the service used, but they link to the service that has additional context.
Story Definition of Ready (updated on 12/23/22)
The following criteria must be met in order for the user story to be picked up by the Flexion development team.
The user story must:
[ ] Is framed in business/user need, the value has been addressed.
[ ] Includes acceptance criteria
[ ] Has been refined
[ ] Pre conditions have been satisfied.
Process:
Flexion developers and designers will test if the story meets acceptance criteria and test cases in Flexion dev and staging environments (“standard testing”). If additional acceptance criteria or testing scenarios are discovered while the story is in progress, a new story should be created, added to the backlog and prioritized by the product owner.
Definition of Done (Updated 5-19-22)
Product Owner
[ ] Acceptance criteria have been met and validated on the Court's migration environment
[ ] READMEs, other appropriate docs, JSDocs and swagger/APIs fully updated
[ ] UI should be touch optimized and responsive for external only (functions on supported mobile devices and optimized for screen sizes as required)
[ ] Interactors should validate entities before calling persistence methods
[ ] Code refactored for clarity and to remove any known technical debt
[ ] Acceptance criteria for the story has been met
[ ] If there are special instructions in order to deploy into the next environment, add them as a comment in the story
[ ] Reviewed by UX on a deployed environment.
[ ] Reviewed by PO on a deployed environment. Can be deployed to the Court's test environment if prod-like data is required. Otherwise deployed to any experimental environment.
As a DAWSON System Administrator, so that I can quickly triage issues in production, I need to be alerted when Lambdas produce errors.
Pre-Conditions
Acceptance Criteria
streams_<ENV>_<COLOR>
api_<ENV>_<COLOR>
api_async_<ENV>_<COLOR>
public_api__<ENV>_<COLOR>
cognito_post_confirmation_lambda_<ENV>_<COLOR>
cognito_post_authentication_lambda_<ENV>_<COLOR>
update_petitioner_cases_lambda_<ENV>_<COLOR>
send_emails_<ENV>_<COLOR>
check_case_cron_<ENV>_<COLOR>
pdf_generator_<ENV>_<COLOR>
seal_in_lower_<ENV>_<COLOR>
cognito_authorizer_<ENV>_<COLOR>
migration_segments_lambda_<ENV>_<COLOR>
migration_lambda_<ENV>_<COLOR>
Additionally, uncaught client side errors should also be reported to a service to help us triage them. This service provides configurable alert rules allowing DAWSON System Administrators the ability to be manage how they are alerted when errors occur. NOTE: It may make sense to have the client side error handling be its own vertical slice.
Notes
Ways to make a Lambda fail
Ways to make Client side errors
Tasks
Test Cases
Lambda throws an Error or logs an error, we would expect the service to receive and store as metadata about the error:
applicationContext.logger.error('message', { contextKey: contextValue })
Errors and exceptions are still logged to the Cloudwatch logs for debugging and troubleshooting
The following errors are not sent to the service:
The service sends email alerts when new errors occur. The information in the emails depends on the service used, but they link to the service that has additional context.
Story Definition of Ready (updated on 12/23/22)
The following criteria must be met in order for the user story to be picked up by the Flexion development team. The user story must:
Definition of Done (Updated 5-19-22)
Product Owner
UX
Engineering
test
environment if prod-like data is required. Otherwise deployed to anyexperimental
environment.staging
environment.