department-of-veterans-affairs / va.gov-team

Public resources for building on and in support of VA.gov. Visit complete Knowledge Hub:
https://depo-platform-documentation.scrollhelp.site/index.html
275 stars 193 forks source link

Add Terms of Use Controller routes monitoring for status #83401

Open bosawt opened 2 weeks ago

bosawt commented 2 weeks ago

We currently monitor Terms of Use Controller application logs, but it appears there might be a use case to monitor TOU controller calls by status as well.

For example, we recently had a spike of 403 (unauthenticated) status in ToU#decline calls, and it would have been good to have a monitor show us that

We should create a series of Datadog widgets that monitor these statuses, I'm thinking something like this: https://vagov.ddog-gov.com/logs?query=%40http.url_details.path%3A%22%2Fv0%2Fterms_of_use_agreements%2Fv1%2Fdecline%22%20%40message_content%3A%22Completed%20%23decline%22%20&agg_m=count&agg_m_source=base&agg_q=%40http.status_code&agg_q_source=base&agg_t=count&analyticsOptions=%5B%22bars%22%2C%22warm%22%2Cnull%2Cnull%5D&cols=host%2Cservice%2C%40http.status_code&fromUser=true&messageDisplay=inline&refresh_mode=sliding&storage=hot&stream_sort=desc&top_n=10&top_o=top&viz=timeseries&x_missing=true&from_ts=1715708068105&to_ts=1715967268105&live=true

joeniquette commented 2 days ago

link to group: https://vagov.ddog-gov.com/dashboard/kc5-ie8-dh2/identity-tou-performance?fromUser=true&refresh_mode=paused&tpl_var_env%5B0%5D=eks-prod&view=spans&from_ts=1717074257719&to_ts=1717088657719&live=false&tile_focus=1433778621971692

@bosawt do you want alerts for these too?