search

department-of-veterans-affairs / va.gov-cms

Editor-centered management for Veteran-centered content.
https://prod.cms.va.gov
GNU General Public License v2.0
99 stars 69 forks source link

SPIKE: Respond to Platform redis key request regarding Facilities keys #19318

Open jilladams opened 1 month ago

jilladams commented 1 month ago

Description or Additional Context

Platform needs product teams to verify the status of Redis keys that are not defined with an expiration in vets-api.

Facilities owns 3 such keys, apprarently, and we need to provide a high-confidence response to Platform on safety of expiring the keys or removing them altogether.

https://dsva.slack.com/archives/CE4304QPK/p1727205902397569?thread_ts=1727205902.397569&cid=CE4304QPK

We have noticed some Redis keys that are not defined in the redis.yml configuration file of the vets-api repository. Additionally, these keys have a TTL (Time to Live) of -1, meaning they are persisting indefinitely in the Redis store. We would like your assistance in identifying where these keys are being set or defined within the vets-api codebase. Moreover, we are trying to understand why the TTL for these keys is set to -1, as this could lead to unnecessary persistence of data in Redis. Could you please provide guidance on where these keys are located and the reasoning behind the TTL configuration?

Facilities related keys

  1. facility-access-satisfaction
  2. facility_dental_service
  3. facility_mental_health

Patrick Vinograd provided some context around history: https://dsva.slack.com/archives/C0FQSS30V/p1727208432823589?thread_ts=1727206911.544229&cid=C0FQSS30V

In an earlier iteration of the facility locator (before LH provided an upstream API and when VA.gov was still etching facility data from an ArcGIS endpoint), we also merged in data from https://www.accesstocare.va.gov/ that was used to power the "veteran satisfaction with wait times" part of a facility detail page. We would periodically pull a bulk download from the access to care system and cache it in redis. There wasn't really any impetus to expire the data.

I believe the dental and mental health namespaces were a similar thing - the source of truth for those data points was different than that where we were getting the rest of the "specialties offered at VAMC" data, so they had to be separately fetched and merged in.

While the dental/mental health and patient satisfaction data is showing on the facility detail pages, my educated guess (please due your own due diligence) is that neither one is coming from vets-api (or if it is, it's included in the LH facilities API at this point), and so those cache entries could be removed entirely.

They would have been fetched by sidekiq jobs under vets-api/app/sidekiq/facilities - you can see the access_data_download.rb, dental_service_reload_job.rb and mental_health_reload_job.rb are still in source control - are they scheduled? If not, safe to clean up the code and the corresponding redis entries. Likewise if they are scheduled but nothing is consuming the data it would be a great tech debt cleanup. And you could potentially also clean up the forward proxy connection to the access to care API endpoint.

Related ticket

Tasks

Acceptance Criteria

jilladams commented 1 month ago

@Agile6MSkinner FYI I don't think we'll be able to answer the platform request without a SPIKE / talking to @omahane and LH about the data migrations (?). I queued this for refinement tomorrow, but may wanna confirm with Michelle that we need to hop right on it, or if it can wait til Sprint 15+.

cc @mmiddaugh

jilladams commented 2 weeks ago

@jilladams I think Patrick Vinograd's PR fixed this. Need to revisit the Slack thread, and confirm if this is ok now or not.