[epic] DR | Code Assessment, Logging, and Monitoring

[saderagsdale]

Value Statement

As a software developer working on an online application I want to be able to assess the health of our codebase So that I can ensure that the application is maintainable, scalable, and performs optimally.

---

Background Context

When we start working on a new feature or bug fix, we need to be confident that the code we're working with is healthy and won't cause any unforeseen issues or technical debt. To do this, we need to be able to assess the health of the codebase regularly, identifying any areas that require attention or improvement. This means we need access to tools that can help us identify code smells, performance bottlenecks, and other potential issues.

We want to be able to use industry-standard tools and techniques to assess the health of our codebase, such as static code analysis, unit testing, and code reviews. We want to be able to easily track the progress of our code health efforts over time and collaborate with others efficiently to address any issues that arise.

Ultimately, by having a healthy codebase, we can be more productive and efficient in our work, delivering high-quality features and bug fixes that meet and exceed our users' expectations.

Acceptance Criteria

• [ ] We have prioritized and/ or resolved bugs using a Cost of Delay framework.
• [ ] We have setup logs to help us trace the cause of failures
• [ ] We have a way to get notified about issues that occur with the forms
• [ ] We have a cadence for checking and reporting
• [ ] If there are other enhancing changes we could implement, they are noted here for team discussion

Tasks

• [ ] Conduct in-depth code inspection
• [ ] Conduct error investigation using Sentry and Grafana logs
• [ ] Learn about how Data Dog handles error states
• [ ] Test drive the Logging and Monitoring Framework, and report back to Jared/Kevin

Definition of Ready

• [ ] Tasks for the assessment of the codebase are defined, (example: static code analysis, unit testing, and code review).
• [ ] Any issues or concerns identified during this assessment have a documentation and prioritization framework.
• [ ] Logging requirements have been identified and documented, including the type of data to be logged, the frequency of logging, and the level of detail required.
• [ ] The necessary logging infrastructure has been set up and tested, including the logging framework and any required configurations.
• [ ] Any dependencies or prerequisites for the epic have been identified and documented.

Definition of Done

• [ ] All identified code smells have been addressed, and the codebase meets agreed-upon coding standards.
• [ ] All identified bugs or issues have been resolved.
• [ ] Unit tests have been created and executed for all critical components.
• [ ] Integration tests have been created and executed to verify that the application functions correctly as a whole.
• [ ] Performance testing has been conducted, and any identified performance issues have been resolved.
• [ ] The logging framework has been implemented and configured according to the documented requirements.
• [ ] All required data points are being logged at the appropriate frequency and level of detail.
• [ ] The logs are being stored in a centralized location and can be easily accessed and analyzed.
• [ ] The logs are being used to monitor the health of the application and identify potential issues proactively.

[saderagsdale]

Phases:

Assessment

Filter queries to reduce noise + identify big issues Review existing logging and monitoring to check for things that were once important Review the form to understand what actions the form can take (list them out) Identify gaps in tracing issues (example: jobs missing from query, downtime from upstream services, add body of errors from upstream services) Flag error messages that aren't actionable (backend or frontend) If possible, recommend fixes for frontend errors to improve user experience KPIs

Craft KPIs for how you measure success for those things Implementation

Determine what you want to see in a dashboard (visuals, alerts, grouping/ungrouping) Add logging points Review with eng (squad 2) Bugs

Prioritization and fixes for anything you've identified

[saderagsdale]

Ticket #1: Identify KPIs for all five of these forms ("assessment") Ticket #2: Set up monitoring and logging for these KPIs if not already set up ("implementation") Ticket #3: Refine sentry error monitoring dashboards for all five forms ("assessment" and "implementation", simultaneously)

Ticket #1 and Ticket #2 must be done in series, and will ultimately result in a series of DataDog dashboards Ticket #3 will ultimately result in a series of Sentry dashboards (the same ones you already saw, just refined) Ticket #3 and Tickets #1-#2 can be done in parallel

[Mottie]

Did we want to include COE in this list?

[saderagsdale]

Nope. That's going to another team in the future. @Mottie

[saderagsdale]

• Add scope for tracking the submission controllers for each form.

[saderagsdale]

• Do we know what the scope of the problem is?
• Are claim submissions/establishments blocked?
• Is the effective date at risk?
• Is there external pressure to resolved?
• Are there external dependencies/teams required to resolve it?
• Are subsets of submissions blocked, or all submissions?
• Does the user need to be notified?
• Does it impact the user experience?
• Who is impacted? (Veteran, internal user, etc.)
• How many are impacted?
• Effort to resolve?

[va-albers]

"Conduct error investigation using Sentry and Grafana logs" - If we have things we get from Grafana that we can't find in other monitoring systems we should note them to make sure they are carried over (since Grafan is being deprecated).

[va-albers]

Thanks @saderagsdale this looks great. Do you think there are parts of this work that should be built into the template for future development work, and what should that look like? Would that belong as part of this Decision Review, or in it's own separate issue?

[saderagsdale]

@va-albers Hey Steve! Good question. Ultimately we want to make the framework repeatable, so the goal is to use this as a pilot to identify what should be in the template. What that looks like is best defined by developers (so it's intuitive for them). Could be template issues that get stored in Github and become part of the onboarding/collab cycle for engineers.

[data-doge]

High-level overview of roadmap, to discuss at refinement: