Establish SLOs' for Stability and Performance of Vets API

[andreahewitt-odd]

Problem Statement

Stability and performance of vets-api gets severely impacted when external services (such as BGS or EVSS) do not behave as expected. This can occur with any of the external services but tends to be a more severe problem with services that we have no governance over, are unowned and/or have a of lack institutional knowledge and expertise behind them. We have done some sleuthing around this, and will continue, with indicators looking at improvements at both an internal and external component level.

User Impact

Platform teams:

• Time required to troubleshoot latency issues takes time away from maintaining the platform and adding new features
• Time is spent resolving urgent technical issues during incident response, when those issues could better be resolved at global level or by individual VFS teams

VFS teams:

• Global stability of vets-api affects stability of VFS owned vets-api endpoints
• Lack of visibility for global vets-api status affects VFS teams ability to troubleshoot their vets-api endpoints

Veterans/VA.gov users:

• Slowed load times across website
• Services (login, form submissions, etc) failing due to timeouts of responses from vets-api

Where was this problem reported?

• Support channel
• Incidents
• Postmortems

How well do we understand the problem?

• There are specific examples which are well understood and have some epics already open around it. FE caching of feature flags, latency epic
• Some discovery needed about broader stability issues
• There are not currently defined SLIs and SLOs for Platform owned vets-api endpoints

What is the acceptance criteria?

• [x] SLIs and SLOs are proposed for Platform owned vets-api endpoints
• [x] Proposed SLIs and SLOs are approved by OCTO-DE Platform
• [x] Monitors and dashboards are created for SLIs
• [x] Dashboard is created for SLO summary
• [x] Alerts are triggered at SLO thresholds
• [x] Internal documentation exists for Platform vets-api SLOs Draft Here
• [x] External (VFS facing) documentation for Platform SLOs and dashboard exists on Platform Website

How should we measure success?

• Number of incidents where vets-api SLO is not met
• Ability of VFS teams to see Platform vets-api status through dashboards
• Have we met SLO for this week/month/quarter

TODOs

• Convert this issue to an epic
• Assign project team name
• Identify this project's OCTO and Vendor key POCs
• Identify specific VFS teams to be part of this project's acceptance
• Identify project target end date
• Conduct project kickoff meeting
• Develop and implement project communication plan

[mchelen-gov]

• Updated success measurements to specify metrics.
• Removed feature toggles and redis references from AC and success measurements, as they are only relevant if impacting other AC & metrics.

[mchelen-gov]

We should make a distinction between stability & performance across all API endpoints (including VFS) and stability & performance of Platform owned API endpoints.

For example: api.va.gov/some/vfs/endpoint - Target is <= 15 sec api.va.gov - Target is <= 2s

[andreahewitt-odd]

Problem statement should be around stability and performance without the context of what's causing it.

[andreahewitt-odd]

break up impact into VFS, Platform and VA.gov users

[raywangoctova]

https://vagov.ddog-gov.com/dashboard/uhg-yyw-yyv/erics-dashboard-mon-aug-22-31427-pm?from_ts=1663691796956&to_ts=1663695396956&live=true

[mchelen-gov]

Problem statement: Overall stability of vets-api, external services are likely cause

User Impact: Reflect significance of each group

Where was this reported? Support channel Incidents Postmortems

How well do we understand the problem?

• Created based on specific examples which are well understood
• Some discovery needed about broader stability issues
• Lack of understanding for SLIs and SLOs

AC

• Propose SLIs (service level indicators) for agreement by OCTO
  • vets-api SLIs are documented
• Tracking, monitoring, and dashboard visibility for SLIs
• Implement OCTO provided SLOs (service level objectives)
  • vets-api SLOs are documented
• Alerting is implemented for SLO thresholds
• Alerts are valid and actionable

Success criteria

• How closely do SLIs match SLOs
• Percentile response times
• Percent of incidents that were proactively alerted on

TODO:

• Post current dashboard screenshot and links

[mchelen-gov]

https://vagov.ddog-gov.com/dashboard/uhg-yyw-yyv/erics-dashboard-mon-aug-22-31427-pm?from_ts=1663691796956&to_ts=1663695396956&live=true

Ok what I'm seeing here is some SLIs that are definitely applicable:

• average api.va.gov response time (synthetic)
• average api.va.gov response time by url (synthetic)

There are also metrics which are meaningful for debugging but not really representative of user impact, such as:

• Puma backlog
• Puma pool capacity

And some metrics which are relevant but not within Platform scope:

• External API status
• Breakers tripped

The acceptance criteria reflects the goal of having mutually understood and defined Platform SLIs and SLOs for vets-api which are documented and visible in a single dashboard.

[mchelen-gov]

References for SLI and SLO definition in increasing levels of detail:

• https://cloud.google.com/blog/products/devops-sre/sre-fundamentals-sli-vs-slo-vs-sla
• https://cloud.google.com/blog/products/management-tools/practical-guide-to-setting-slos
• https://sre.google/workbook/implementing-slos/
• https://sre.google/workbook/slo-document/

[annekerr49]

https://app.zenhub.com/files/133843125/91f4ee9a-797c-4dff-ad90-524cde55ff4f/download @andreahewitt-odd

[mchelen-gov]

Some discovery needed about broader stability issues

Can anyone clarify what this means?

[mchelen-gov]

@little-oddball It looks like you accidentally overwrote the text of the issue in your last edit, so I have restored previous version.

[raywangoctova]

Note from Platform Leadership FY23 planning onsite: vets-API latency: decomposition on endpoints that will benefit from isolation and scalability.

[annekerr49]

@ericboehs Eric do you have any ideas about what milestone we should attach to this product?

[npeterson54]

Removing first AC into its own criteria, story created here

[mchelen-gov]

Given the revised scope, here are the remaining tasks needed for completion.

• [ ] Create single dashboard for Platform SLOs
  • Each SLO is displayed
  • Only metrics or monitors which directly feed into an SLO are displayed
  • Draft derived from vets-api dashboard
• [ ] SLOs are named consistently within Datadog
  • Examples of inconsistent naming:
  • "[SLO] Platform - Puma Capacity (Secondary ASG)"
  • "Platform - Puma Capacity (Primary ASG)"
  • "Availability SLO on vets-api service"
  • [ ] SLOs, SLIs, and monitors are tagged consistently within Datadog
  • env, service, and team tags exist for each item
  • inconsistent tags such as "env:prod" vs "env:production" are reconciled
  • [ ] SLOs and SLIs are named consistently between documentation and Datadog
  • "capacity" vs "saturation"
  • "Puma Capacity" vs "Puma Backlog" vs "puma_backlog"
• [ ] Datadog monitors have descriptive names
  • Generic descriptions such as "[Synthetics] Test on api.va.gov" should be revised
• [ ] Documentation links SLIs and SLOs in Datadog
  • Confluence doc directly links to items in Datadog for each SLI and SLO described
• [ ] Documentation is published on Platform website
• [ ] SLO and dashboard are launched and announcement is made

[jwoodman5]

Hey @mchelen-gov, With the recent pivot for this team and some of the team members in general, can you please clarify if you have a thought on when the remaining work would happen and who would be doing it?

[mchelen-gov]

Hey @mchelen-gov, With the recent pivot for this team and some of the team members in general, can you please clarify if you have a thought on when the remaining work would happen and who would be doing it?

This project's current scope should be wrapped up before any team pivots. Let's discuss more on Monday.

[jwoodman5]

@mchelen-gov I sync'd with Nate today on the plan y'all discussed yesterday. Sounds good.

[BillChapmanUSDS]

Request to post SLO doc to pltform website filed here: https://github.com/department-of-veterans-affairs/va.gov-team/issues/56821

[acrollet]

Comment and a question:

• looks like the SLOs for vets-api have been posted
• is there a corresponding issue for VA.gov/vets-website SLOs? I dug around but couldn't find one.

[annekerr49]

No matches for roadmap-DMC

[annekerr49]

close