Tech Spec: Get data from va.gov into VRO S3 bucket

[dianagriffin]

User Story

As a VRO team member, I want to understand our solution and technical approach for accessing and retrieving va.gov activity data related to the 526 form, so that I can contribute to implementing the solution.

Acceptance Criteria

• Solution and technical approach for retrieving data from va.gov's Postgres database and putting it into an S3 bucket in VRO is documented and shared with the team and @CorySohrakoffUSDS, including a breakdown of tasks
  • Tech spec should identify: technical solutions considered, recommended technical solution (with pros and cons), assumptions and open questions underlying our recommended solution, dependencies, and a (draft) breakdown of tasks to accomplish the technical solution
• The VRO team is aligned on the technical solution and has a shared understanding of the task breakdown, enabling implementation of the solution to begin

Context

• Problem overview
• High-level solution diagram
• Known data needs/questions
• Related tickets:

  • 1923

  • 1849

[tejans24]

• What are the list of data questions to answer the MVP for Benefits portfolio?
• Rails console access, what are the steps needed to gain rails console access?
• Understanding the va.gov codebase, see where a job to fetch database data can be added to the repo
  • Setup vets.api locally, run it to create a development environment (including the sidekiq/jobs)
  • Research: Does the job (assuming sidekiq) need any additional permissions to access the db?
  • Research: Does auth/access between vets.gov access server to S3 bucket?
    • What level permissions are needed for the script to access S3 bucket to put files into S3 Bucket in LHDI env.
• Write the script that fetches changes in db every X period
  • What is that X period?
  • Understand the ActiveRecord modeling of the datasets needed to query the data to answer the MVP questions.
  • Decrypt encrypted fields to store it as json to s3 bucket
    • Does the raw data from db have fields that need to be decrypted? If so, what is the mechanism to decrypt? Where are the encryption keys located?
  • Error handling of the jobs? How does VRO get notified?
  • Metrics on the jobs (how long do the jobs take, when it was executed, success/failures)
    • Can this be in DataDog? Do we need to send this as metadata to s3?
  • What level permissions needed to setup the cron job schdule?
• Design the database schema in VRO based on the data format provided from the form

[dianagriffin]

LIst of data questions to answer: https://app.mural.co/t/departmentofveteransaffairs9999/m/departmentofveteransaffairs9999/1689805673860/562838f00f27c06b8ff311f6d15da3da839b1bb2?wid=0-1690311565035

[dianagriffin]

@dianagriffin find the thread about rails console access and link here!

[dianagriffin]

What are the steps needed to gain rails console access?

• Slack thread here points to:
  • This request form
  • An example of a submitted request from another team

Based on what I'm seeing in the example submitted request, we may need some guidance from @CorySohrakoffUSDS or input from @verdance-luke and/or @nanotone on specifically what access they requested that allowed rails console access. (For example, Kyle's example requests group_memberships = ["dsva-vagov-developers", "force-MFA"] and "adhoc-vetsgov-ssm-prod" -- the latter sounds like the prod access piece, but also sounds like it's a group specific to another team so we need to figure out if there's an appropriate group for our folks.

[Edit] Hmm, actually, I found the following two tickets from Luke's access requests:

• https://github.com/department-of-veterans-affairs/va.gov-team/issues/64690
• https://github.com/department-of-veterans-affairs/va.gov-team/issues/63689 ^ The first one made me realize there's a separate form to request terminal access: https://github.com/department-of-veterans-affairs/va.gov-team/issues/new?assignees=&labels=external-request%2Cplatform-tech-team-support%2Cops-access-request&projects=&template=vetsapi-argo-terminal-access.yaml&title=Vets-api+terminal+access+for+%5Bindividual%5D

@tejans24 do you have a sense of whether we'd need just terminal access, or also AWS access? Or is that a question we should get input from our friends already working in vets-api?

[dianagriffin]

Write the script that fetches changes in db every X period

• What is that X period?

It's up to us to propose the answer to this. From a product perspective, I'll be pushing for "as often as we reasonably can" -- so the question back to y'all engineers is: what's reasonable?