ITF Failure Dashboard cleanup

[tblackwe]

When the ITF service is down or responds with an error the Veteran is unable to proceed with creating a 526 claim.

To determine how frequently this issue occurs, please review the existing ITF dashboard to ensure they are accurately capturing errors

The dashboard has v0 which is the vets-api controller. v2 is the lighthouse api.

AC:

• [ ] Dashboard fixed to accurately portray success/timeout/error rates
• [ ] Dashboard indicates the application that generated the ITF
• [ ] Alerts created for ITF outage
• [ ] Accurate labels on all the graphs and chart
• [ ] consolidate dashboards

[sortizsh]

Thomas can you validate that this cleanup is still needed?

[freeheeling]

Dashboards for reference:

• Intent To File - Overview
• Benefits - DBex - EVSS-to-LH: Intent to File
• Benefits - Intent to File
• Benefits - ITF Success/Timeout/Error rates

Questions:

1. Why so many 404s?
2. Why don't the widgets have data to display?

Thomas to confirm expectations.

[emilytheis]

1. 404s are a normal/expected response for a GET that fails. However, if POSTs are failing, we should investigate that.

[va-albers]

@freeheeling regarding "Why don't the widgets have data to display?" - I would start by checking if the service name for the controller being queried needs to be changed/removed. That could have resulted in empty fields. For instance I removed the service:vets-api parts of the queries on Intent To File - Overview all all the widgets are showing data now.

[va-albers]

(@freeheeling FYI I updated the 4 dashboards above removing the service:vets-api reference on some widgets - all widgets should show up now)

[tblackwe]

Scheduling meeting with Mark Chae to discuss

[freeheeling]

Thanks, @va-albers. Appears I don't have Dashboards Write permission, but I'm also not clear on where to modify the service name being queried.

[freeheeling]

Responding to Emily's comment from 5/7, on the Benefits - ITF Success/Timeout/Error rates dashboard, there do appear to be 404 status codes from POST requests. However, when viewing the related logs, at least for the few I inspected, the requests are all GET. So, perhaps, the dashboard widget is labeled incorrectly. I'm still a Datadog novice with limited permissions, fumbling around without much guidance.

[va-albers]

Thanks, @va-albers. Appears I don't have Dashboards Write permission, but I'm also not clear on where to modify the service name being queried.

I will check this

[lisacapaccioli]

@va-albers It's been a while since this ticket has been viewed and I am not sure what to do with it. Is this something you can help with? (cc @tblackwe)