Develop a general asynchronous processing model for vets-api requests

[ricetj]

Problem Statement: Many vets-api requests that flow through to various backend systems take a long time to complete. This may be due to processing or data transfer or other unknown performance limitations in legacy systems. POST and PUT requests that submit or update information are frequent offenders, but there are some long-running GET requests as well.

Performing these operations synchronously in the context of the API request has several downsides:

• It risks encountering client timeouts.
• It consumes Rails/Puma threads for a long time, risking thread exhaustion and thus causing concurrent API requests to fail.
• It results in high/uneven aggregate request latency which makes it hard to define resonable service level objectives.

Proposal: Develop a common mechanism for processing long-running requests asynchronously that can be used across vets-api.

• An API convention for representing asynchronous requests, e.g. how are requests identified, what status information is returned during processing/upon completion.
• A common library or development pattern for executing the asynchronous requests and returning appropriate status information as represented in the API.
• Provisions for monitoring request processing effectively even when this pattern is widely adopted.
• Provisions for dealing with failures during asynchronous processing - can jobs be retried/resumed/identified for follow-up.

Existing Patterns: Several form-submission type applications use asynchronous processing or submission.

• The VICv2 application spawns asynchronous workers to scan and re-encode supporting documentation and image content, and to submit ID card applications to Salesforce.
• The Preneed burial application spawns an asynchronous job to submit the completed application to the EOAS backend, and polls for status updates.

Concerns:

• We already use sidekiq for worker jobs. How fine-grained do we want to get with assignment of jobs to queues to enhance monitoring of the system, while staying within the constraints sidekiq imposes. Can we introduce additional tooling to enhance monitoring?
• What is the best-practice API pattern for asynchronous requests?

Questions:

• How does this impact the POST / PUT work hpjaj is working to implement for profile? Should we proceed, and if so, do we have answers to the blockers there? What is our expected timeline either way?

[ricetj]

(03/22/2018) This seems interesting: https://github.com/utgarda/sidekiq-status#retrieving-status

[ricetj]

(03/26/2018) notes from call:

How did this come about?

• Make our API meet industry standards for latency response
• been on platform for a while
• splitting API and background thread so can do asynchronous
• “artificial, only on paper improvement”

This ticket includes current design constraints Best practice examples of asynchronous API for frontend exist out there

• Can we find a best practice to implement (if rainbows didn’t already build it with VIC - http request, controller)

• Think through backend semantics for handling errors, reporting it back, do we retry automatically? Manually? ^ hpjaj and emersonthis - can we think through this

• patrickvinograd suggests syncing with rlbaker on this since he built out relevant backend (cc rtluu)

PCIU telephone suggested as starting use case - steve-gov

Blockers on post/put ^decision on what to do approach wise - this call resolved ^if access is not granted with the GET, that would be a blocker

Goals:

1. Show our API has better latency, can push on partners to improve backend latency
2. Don’t want synchronous processing of API
   • currently the case, leads to thread exhaustion
3. Create consistent API implementation

[ricetj]

(03/28/2018) I have some questions around reliability and monitoring. I don't think we need to nail all of these down before moving forward with a design, but we'll want to be keeping them in mind as we get closer to a production deployment.

• What durability guarantees does sidekiq provide for queued jobs?
• What is the backing store for queueing jobs?
• Do sidekiq jobs have automatic retry capabilities?
• How many concurrent tasks does our instance allow?
• What is the queue limit on queued jobs?
• What happens if that queue limit is exceeded?
• How will circuit-breaker pattern integrate with job retries?
• Can a user cancel a pending update?
• Can a user have more than one pending job queued / running concurrently?
• Can an administrator cancel one or many jobs to mitigate an outage?

Sorry if the answers to some of these are obvious, but sidekiq is not a technology with which I'm familiar.

Some guarantees I'd like to see:

• If an async job is enqueued, it survives a process or server crash (durability)
• If the system is over capacity (queue limit exceeded), the user sees an immediate error
• If a service that vets.gov-api depends on is down, we avoid a "thundering herd" when it comes back online

[ricetj]

(03/28/2018) This is shaping up to be a bit more of an endeavor than we (or I?) had originally anticipated.

I wonder if perhaps we should decouple the creation of the new endpoints from the discovery & assembly of this new asynchronous request track. I feel these things can be done in parallel.

I raise this idea because these last few endpoints are the final pieces of the puzzle for the Profile form, and am concerned that this will add too much of a delay to the form's release.

In light of that, I am proposing that:

1. These new POST endpoints for the profile form follow a standard synchronous request pattern
2. Convert this story into a discovery story (or create an associated one), that answers all the questions raised thus far, as well as the ones we have not thought of yet
3. Based on the deliverable from that discovery story, create any sub-issues to enact a standalone asynchronous request handler

What are everyone's thoughts?

[ricetj]

(03/28/2018) think this is very smart and agree wholeheartedly at this stage

This was my biggest concern with taking on asynch approach coupled to the POST/PUT endpoint delivery so late in our process, with so little to finish on profiles backend for launch.

I think it’s very smart to separate the smaller, more immediate launch task from the asynch discovery

[ricetj]

other notes can be found here, not moved in case not suitable for public repo: https://github.com/department-of-veterans-affairs/vets.gov-team/issues/9400