Enhance 526 submission metrics by tracking service provider

[pacerwow]

Issue Description

As a DBEX or OCTO team member, I want to be able to easily identify the submission performance of the Lighthouse migration compared to EVSS. This includes use of the primary, backup, and failure rates viewable as a time series.

We want to understand the following: Primary path success (evss) Primary path failure (evss) Backup path success (evss) Backup path failure (evss) Primary path success (lighthouse) Primary path failure (lighthouse) Backup path success (lighthouse) Backup path failure (lighthouse)

---

Tasks

• [x] Create a Datadog dashboard to house the metrics we'll gather
• [x] Track which service provider is used when calling the Submit endpoint in either EVSS or Lighthouse
• [x] Identify the graphs using plain language so that any team member can understand what they are looking at
• [x] Demo the dashboard with DBEX and OCTO during Sprint Review

Acceptance Criteria

• [x] We will be able to understand the paths listed above in a place that is easily shareable and understood
• [x] Performance and functionality of the Lighthouse and EVSS services used for the 526 form are not worse than before this change

---

How to configure this issue

• [ ] Attached to a Milestone (when will this be completed?)
• [ ] Attached to an Epic (what body of work is this a part of?)
• [ ] Labeled with Team (product support, analytics-insights, operations, service-design, Console-Services, tools-fe)
• [ ] Labeled with Practice Area (backend, frontend, devops, design, research, product, ia, qa, analytics, contact center, research, accessibility, content)
• [ ] Labeled with Type (bug, request, discovery, documentation, etc.)

[kylesoskin]

Can we make sure at least every event from this comment/table, has logging setup, with appropriate info in each log message?

Logged data

• User_uuid
• Submission id and/or saved_claim_id
• Action being performed (Form ID or action description)
• Up or Downstream system involved (EVSS or Lighthouse, base logging from the api provider classes)
• If a retriable action -- Attempt counter -- If will be retried or if it is the final attempt
• Success or failure status
• Http status from up/downstream service
• Http response body if NOT a 2XX (successful) response (and assuming it has no PII, it shouldnt.....)
• Any ID that is the result of a creation, update or delete (IDs returned from third-party services, internal VA.gov DB record IDs)
• Exception messsage/stacktrace, if fails outside/before/after external call

Metrics

• Incremental
• Success
• Failure
• Total (Or any 2 of the 3 that allow you to arrive at the 3rd)
• Overall Retry counter (if applicable)

[mchae-nava]

Dashboard link https://vagov.ddog-gov.com/dashboard/2aa-hng-k4s?fromUser=false&refresh_mode=sliding&tpl_var_env%5B0%5D=eks-prod&from_ts=1729004144017&to_ts=1729608944017&live=true