SPIKE: Become familiar with the facility locator data architecture

[kmariepat-cityfriends]

Background

Refer to this card: https://app.zenhub.com/workspaces/sitewide-facilities-639f5253e4b702a32376339e/issues/gh/department-of-veterans-affairs/va.gov-cms/12984

Acceptance Criteria

• [x] I expect research is done to confirm if there is any existing documentation on facility locator data architecture: https://github.com/department-of-veterans-affairs/va.gov-team/blob/master/products/facilities/facility-locator/issue-response.md

• [x] I expect that the information learned is knowledge shared with the rest of the team

• [x] I expect this card results in a document of the facility locator data architecture, that we confirm is representative of the current state which should include all facility locator data sources, see below:

• [x] I expect that existing documents are updated to be accurate, if/where they are inaccurate. (e.g. diagrams that are incorrect can be updated using Mermaid. @swirtSJW SME can help with add'l info.)

• vets-api

• ppms

• lighthouse-facilities api

• va.gov-cms api

[kmariepat-cityfriends]

Dave C. brought up existing documentation that Lance completed within the facility locator GitHub repo https://github.com/department-of-veterans-affairs/va.gov-team/tree/master/products/facilities/facility-locator/engineering, this should be reviewed on the spike

[jilladams]

From 1:1 discussion with Max:

• Blocked = blocked on seeding data locally from what exists already in vets-api-mockdata
• Support request opened https://dsva.slack.com/archives/CBU0KDSB1/p1679944111130949, but got no traction
• Max noted the instructions for the process are vague. Josh / Daniel have also had issues here.
• Max will follow up and try to get a response.

[maxx1128]

I am catching up after last week and am adding relevant notes here as I go over the repo again.

The documentation points to controllers/v1/facilities/va_controller.rb for the facilities data endpoint, but the code that sorts out those responses is in lib/lighthouse/facilities/. These are the files that take in the responses from the Lighthouse Facilities API, sort out the properties into the right format, and add extra properties like min and max time for nearby ones. So I anticipate that most of the changes we make for the facility locator front-end would be here.

• facility.rb is the base model of a piece of facility data. For Ruby on Rails, a model is basically the structure of a piece of data, including all the attributes' names and their data types. Changes to the amount or types of data included in a facility will need to be made here.
• response.rb is the broader response from the Lighthouse API, which includes any facilities from the API as well as metadata like the response status or number of results pages. The data for the facilities themselves are created by facility.rb
• nearby_response.rb and nearby_facility.rb add functionality on top of response.rb and facility.rb
• client.rb is what makes the actual request to the Lighthouse API and returns either response.rb or nearby_response.rb.

[maxx1128]

Changes in the data attributes would likely be in app/serializers/lighthouse/facilities/facility_serializer.rb. This is the file in charge of formatting all the data into a predictable, functional format. Plus it defines the basic facility attributes, so changes in those would likely need to be reflected here too.

[maxx1128]

lib/facilities/ files are ones that are not Ruby on Rails specific and are mostly static Facilities data or base configuration. It's unlikely these will ever be changed for our changes. But if there are ever vague or weird errors with the API requests, it's worth checking if the files here are related to them.

[maxx1128]

For general Ruby knowledge that would help with navigating vets-api, I would recommend these Github documents on some Ruby fundamentals, Methods, and Structs. This is also a helpful introduction to object-oriented programming and Ruby.

For general Ruby on Rails knowledge sharing, I think the most important part of navigating the codebase and finding needed functionality is the folder structure. All the below files are in the app/ folder with a fast breakdown of each.

• controllers are the files that are used to handle client web requests. URLs are defined in config/routes.rb, and this file also decides what controller will handle it. One controller will usually handle the different request types such as "get" and post." They will often also link out to other Ruby classes as needed to handle different functionality as needed (for example, calling a model to pull up related data).
  • For example, claims_base_controller.rb is for web requests related to claims. This includes what's needed to create a claim and show a specific claim.
• mailers contain all functionality for mailing jobs, such as emails for password resets or newsletters. The views folder inside it contains the HTML templates used for the emails.
  • For example, dependents_application_failure_mailer.rb is what sends emails when a dependency claim application has failed and alerts the user to this.
• models are how specific pieces of data are defined and structured. If something is saved as a row of data in a database, it is defined by a model. Models control the names and qualities of each data attribute, other data it's associated or connected with, any custom properties, and validations to run before it's saved in a database.
  • For example, bank_name.rb defines how each bank name is saved. It shows that each one has a bank_name and routing_number property saved as a string, makes sure each bank name has a routing number and also a method that defines the bank name if it is empty but a routing number is present elsewhere in our data.
• policies are small classes that define more complex logic for reading properties off existing objects.
• serializers are for quickly restructuring data objects into more predictable/convenient/readable formats. They may also define some base object structure for data that may be important but not represented with a model.
• services are for more complex operations for reading and/or writing data that involve multiple steps and juggling multiple classes.
• uploaders are for operations related to uploading different file types and related operations, such as virus scans or validating PDFs.
• validators are for any custom validator operations outside of models
• workers are classes run through sidekiq meant to run in the background as needed or at set time intervals. For example, there's one that routinely gets rid of removed flippers.

Info on other (custom) Rails patterns can be found here.

[maxx1128]

This is a small overview of some of the other important folders in the Rails app, outside of the app/ folder.

• bin/ is for scripts run from the command line. They're usually for setup and running the app locally, and run through the command line with bin/<file_name>.
• config/ is for all app configuration, including environmental variables. routes.rb is also here for writing what URLs go to which controllers.
• db/ for for database-related files, like the initial seed data and any migrations.
  • Migrations are files that create needed changes to the database table structure, like updating/adding/removing columns or adding new tables. Migrations must always be up to date before running locally.
• lib/ is a large catch-all for app functionality or data that doesn't fall into typical Rails conventions or folders. This includes code related to third-party modules, like lighthouse and its related facilities endpoints.
• public/ are for any files that are directly visible to public pages or endpoints, like error pages or image assets.
• `spec/' is for test files

[jilladams]

@maxx1128 https://github.com/department-of-veterans-affairs/va.gov-team/blob/master/products/facilities/facility-locator/engineering/README.md. FYI

And https://github.com/department-of-veterans-affairs/va.gov-team/tree/master/products/facilities/facility-locator/data

[maxx1128]

From what I'm reading and re-reading in these documents, the basic data architecture and setup is mostly lining up with what I see in the code. For a facility locator data request, the process basically goes like this:

• vets-api receives a request for facility locator data (at app/controllers/v1/facilities/va_controller.rb).
• That request for data has to come from our lighthouse API endpoint. The vets-api repo sends its own request to this endpoint (at lib/lighthouse/facilities/client.rb).
• There are three basic types of data requests this repo makes for facility data:
  1. Getting details of one facility by its specific ID (at lib/lighthouse/facilities/response.rb)
  2. Getting a list of facilities that meet a specified set of parameters (at lib/lighthouse/facilities/response.rb)
     • This includes an option of excluding mobile facilities
  3. Getting a list of nearby facilities that meet a specified set of parameters (at lib/lighthouse/facilities/nearby_response.rb)
     • This does not include an option of excluding mobile facilities
• All the facility data is then formatted and prepared through a serializer (at app/serializers/lighthouse/facilities/facility_serializer.rb)

I don't see the PPMS coming into play at any point in this process, which may be the one difference I see between the code and a few of the documents. But otherwise it looks accurate.

The data architecture of each piece of facility data is as follows:

# "access" is initially under the "wait_times" attribute
attribute :access, Object

attribute :active_status, String
attribute :address, Object
attribute :classification, String
attribute :detailed_services, Object
attribute :distance, Float
attribute :facility_type, String

# This is the first part of the "id" attribute, before the "_" symbol
attribute :facility_type_prefix, String

# "feedback" is initially under the "satisfaction" attribute
attribute :feedback, Object

attribute :hours, Object
attribute :id, String
attribute :lat, Float
attribute :long, Float
attribute :mobile, Boolean
attribute :name, String
attribute :operating_status, Object
attribute :operational_hours_special_instructions, String
attribute :phone, Object
attribute :services, Object
attribute :type, String

# This is the second part of the "id" attribute, after the "_" symbol
attribute :unique_id, String

attribute :visn, String
attribute :website, String

The attributes that are Object have nested attributes within them, but they're not specified within the code.

[maxx1128]

I've been looking more into the PPMS code and it looks like it lines up with this documentation about the endpoints it hits. This is a dev-focused overview of what files are hit when making a PPMS API request.

1. The request starts when it hits the controller modules/facilities_api/app/controllers/facilities_api/v1/ccp_controller.rb.
   • There's another at modules/mobile/app/controllers/mobile/v0/community_care_providers_controller.rb but that looks like it's no longer use since it's with the older API version.
   • The controller handles requests for these four types of facilities data: urgent care, provider, pharmacy, and specialities. This lines up with the documentation.
2. All of the above four data requests go to modules/facilities_api/app/services/facilities_api/v1/ppms/client.rb. This is what starts the process of making the API request and returning it in workable data.
3. In modules/facilities_api/app/services/facilities_api/v1/ppms/client.rb, there is a different method for each of the four request types. Each one references a different endpoint PPMS URL to get the needed data (listed as private methods in the class).
   • This service pulls in configuration data from modules/facilities_api/app/services/facilities_api/v1/ppms/configuration.rb, which has the code that makes the actual HTTP connection to the API endpoint in the connection method.
     • It uses the Faraday gem for making the request and getting the response
     • modules/facilities_api/app/services/facilities_api/v1/ppms/middleware/ppms_parser.rb parses the raw API response
   • This file holds details on the request parameters, like pagination, search radius, and the number of results.
4. When the API response has been completed and initially parsed, it is then passed to modules/facilities_api/app/services/facilities_api/v1/ppms/response.rb. This file converts the response into a data model, which holds information on all the attributes each one has.
   • All of the four data types are passed to the same model file: modules/facilities_api/app/models/facilities_api/v1/ppms/provider.rb. The exception is again specialities, which instead uses modules/facilities_api/app/models/facilities_api/v1/ppms/specialty.rb. These data files are in charge of managing how the response data is handled by other Ruby code (data types, data defined through separate methods, cleaning up data, and see all the basic attributes). Information on PPMS data structures can be found in these two files.
   • This response file also does the actual pagination based on past parameters. This is with the will_paginate ruby gem, which is designed for these kind of ActiveRecord data objects.
     5. These ActiveRecord data objects then come back to modules/facilities_api/app/controllers/facilities_api/v1/ccp_controller.rb, finished and in a Ruby-friendly form. They are all piped through modules/facilities_api/app/serializers/facilities_api/v1/ppms/provider_serializer.rb - the exception is specialties, which go through modules/facilities_api/app/serializers/facilities_api/v1/ppms/specialty_serializer.rb. These convert the data into a JSON format, which is what is ultimately given to whoever is making the API request.
   • The serializer makes sure any nested objects in the data are converted to JSON properly. This also handles renaming them and saving values under multiple names. Additional information on PPMS data structures can be found in these two files.

[maxx1128]

This is the data architecture for the different PPMS data types.

Urgent Care, Providers, and Pharmacies

# modules/facilities_api/app/serializers/facilities_api/v1/ppms/provider_serializer.rb

attribute :accNewPatients

attribute :address # the below four attributes are only included if they all exist. Otherwise it's only the address
    :street
    :city
    :state
    :zip

attribute :caresitePhone
attribute :email
attribute :fax
attribute :gender
attribute :lat, :latitude
attribute :long, :longitude
attribute :name
attribute :phone, :mainPhone
attribute :posCodes # if there are multiple, it only shows the first
attribute :prefContact, :contactMethod
attribute :uniqueId, :providerIdentifier

Specialties

# modules/facilities_api/app/serializers/facilities_api/v1/ppms/specialty_serializer.rb

attribute :specialityCode # is the unique ID
attribute :classification
attribute :grouping
attribute :name
attribute :specialization
attribute :specialityCode
attribute :specialityDescription

[eselkin]

Just some related info, to install currently on a mac from a fresh homebrew with new rvm

brew install shared-mime-info
# not full postgresql install, just the lib
brew install libpq 
# because libpq by itself links not in a normal location for gem/ruby
gem install pg -- --with-pg-config=/opt/homebrew/opt/libpq/bin/pg_config 
bundle install

You can run the system pretty easily with

make up

This startup starts redis/postgres DBs and the vets-api You have to do a

make build
or 
make rebuild

prior to make up

Can't currently get LIGHTHOUSE_API_KEY