Errors trigger alarm notifications

[mmarcotte]

As a DAWSON System Administrator, so that I can quickly triage issues in production, I need to be alerted when Lambdas produce errors.

Pre-Conditions

Acceptance Criteria

• Errors continue to be logged to Cloudwatch logs via the Application logger.
• Errors and exceptions are also reported to a service that helps us triage and address them, giving us an easy way to connect them to github issues and the deployments they were released.
• Lambdas that need to be included:
  • streams_<ENV>_<COLOR>
  • api_<ENV>_<COLOR>
  • api_async_<ENV>_<COLOR>
  • public_api__<ENV>_<COLOR>
  • cognito_post_confirmation_lambda_<ENV>_<COLOR>
  • cognito_post_authentication_lambda_<ENV>_<COLOR>
  • update_petitioner_cases_lambda_<ENV>_<COLOR>
  • send_emails_<ENV>_<COLOR>
  • check_case_cron_<ENV>_<COLOR>
  • pdf_generator_<ENV>_<COLOR>
  • seal_in_lower_<ENV>_<COLOR>
  • cognito_authorizer_<ENV>_<COLOR>
  • migration_segments_lambda_<ENV>_<COLOR>
  • migration_lambda_<ENV>_<COLOR>

Additionally, uncaught client side errors should also be reported to a service to help us triage them. This service provides configurable alert rules allowing DAWSON System Administrators the ability to be manage how they are alerted when errors occur. NOTE: It may make sense to have the client side error handling be its own vertical slice.

Notes

• Should help prevent silent errors like the ones discovered in Bug 10072
• Ref: #10043
• Perhaps a spike is warranted to determine the best error error handling and monitoring service. The court's current preference is Sentry, and appreciate that there has already been experience on the team with Honeybadger.

Ways to make a Lambda fail

• breaking the code so that it throws an error
• make an entity fail validation by updating the record in the database

Ways to make Client side errors

• breaking the code so a JavaScript error is thrown

Tasks

Test Cases

1. Lambda throws an Error or logs an error, we would expect the service to receive and store as metadata about the error:

   • a stack trace of the error, if one is available
   • any included context that is passed along via applicationContext.logger.error('message', { contextKey: contextValue })
   • the lambda that triggered the error
   • the requestId of the lambda that triggered the error
   • any requests that involve a password or tokens do not have that information included

2. Errors and exceptions are still logged to the Cloudwatch logs for debugging and troubleshooting

3. The following errors are not sent to the service:

   • NotFoundError
   • UnkonwnUserErrror
   • UnauthorizedError
   • UnprocessableEntityError
   • ServiceUnavailableError

4. The service sends email alerts when new errors occur. The information in the emails depends on the service used, but they link to the service that has additional context.

Story Definition of Ready (updated on 12/23/22)

The following criteria must be met in order for the user story to be picked up by the Flexion development team. The user story must:

• [ ] Is framed in business/user need, the value has been addressed.
• [ ] Includes acceptance criteria
• [ ] Has been refined
• [ ] Pre conditions have been satisfied. Process: Flexion developers and designers will test if the story meets acceptance criteria and test cases in Flexion dev and staging environments (“standard testing”). If additional acceptance criteria or testing scenarios are discovered while the story is in progress, a new story should be created, added to the backlog and prioritized by the product owner.

Definition of Done (Updated 5-19-22)

Product Owner

• [ ] Acceptance criteria have been met and validated on the Court's migration environment
• [ ] Add scenario to testing document, if applicable (https://docs.google.com/spreadsheets/d/1FUHKC_YrT-PosaWD5gRVmsDzI1HS_U-8CyMIb-qX9EA/edit?usp=sharing)

UX

• [ ] Business test scenarios have been refined to meet all acceptance criteria
• [ ] Usability has been validated
• [ ] Wiki has been updated (if applicable)
• [ ] Story has been tested on a mobile device (for external users only)

Engineering

• [ ] Automated test scripts have been written, including visual tests for newly added PDFs
• [ ] Field level and page level validation errors (front-end and server-side) integrated and functioning
• [ ] Verify that language for docket record for internal users and external users is identical
• [ ] New screens have been added to pa11y scripts
• [ ] All new functionality verified to work with keyboard and macOS voiceover https://www.apple.com/voiceover/info/guide/_1124.html
• [ ] READMEs, other appropriate docs, JSDocs and swagger/APIs fully updated
• [ ] UI should be touch optimized and responsive for external only (functions on supported mobile devices and optimized for screen sizes as required)
• [ ] Interactors should validate entities before calling persistence methods
• [ ] Code refactored for clarity and to remove any known technical debt
• [ ] Acceptance criteria for the story has been met
• [ ] If there are special instructions in order to deploy into the next environment, add them as a comment in the story
• [ ] Reviewed by UX on a deployed environment.
• [ ] Reviewed by PO on a deployed environment. Can be deployed to the Court's test environment if prod-like data is required. Otherwise deployed to any experimental environment.
• [ ] Deployed to the Court's staging environment.

[mmarcotte]

Blocked by Court Procurement on the service.

[codyseibert]

I'm just going to place this on blocked until next week when we can have a discussion around this.