Remove all API tokens, certificates, and other confidential information from VRO GitHub code.

[va-albers]

User Story

As a LH support team member, I want to ensure that all security tokens, passwords, keys, and certificates are not included in source control, so that we minimize unnecessary security risks to the platform.

Acceptance Criteria

1. Catalog and document all keys, token, certs, etc. across environments, including where they are located (and how they got there), as well as expiration dates and rotation/renewal instructions as applicable (see comments on #1173 for inputs). Note any source code that identifies any current confidential information.
2. Once AC1 is complete, set up a discussion with the VRO engineers, Steve Albers, and Cory Sohrakoff to discuss whether each item needs to be moved elsewhere (and if so, what is the target location).
3. Depending on the outcomes of discussion from AC2, the following items may be determined in scope for this ticket, or may be split into separate follow-up tickets:
   • Any keys, tokens, or certificates determined necessary to remove are removed.
   • Any keys, tokens, or certificates determined necessary to rotate have been rotated, so historical commits are of limited value.
   • If necessary, an Architectural Review Document covering the management of secrets is written, reviewed with the team, and published.

Not included in this work Any secrets that relate to access to PII or production systems - those would be handled through the incident process.

[yoomlam]

Does this include keys, tokens, or certificates for testing and development? Or is this only for prod and prod-test environments?

[va-albers]

Does this include keys, tokens, or certificates for testing and development? Or is this only for prod and prod-test environments?

It includes all environments. If we have specific cases where this is prohibitively complex we should talk about it, but in the general response from LHDI was for all environments.

[yoomlam]

This work involves:

• cataloging all the keys, tokens, certs, etc. and where they are located (and how they got there)
• discussing whether each needs to be moved elsewhere (and if so, what is the target location)
• doing the move (and probably rotation to use new secrets)

[dianagriffin]

We think a good next step on this would be to understand the threat model / do some analysis of the risk.

Noting from discussion that git history is not a risk in our case because the credentials were rotated when committed to git.

[dianagriffin]

This work involves:

• cataloging all the keys, tokens, certs, etc. and where they are located (and how they got there)
• discussing whether each needs to be moved elsewhere (and if so, what is the target location)
• Depending on outcomes of discussion -- doing the move (and probably rotation to use new secrets) if the requirement for what to move is small. If the scope of the move needed is large, we will spin out a new ticket to cover it.

[dianagriffin]

@tcraghu cc/ @yoomlam I have updated the ticket description & acceptance criteria to reflect the steps we aligned on in our refinement sessions and sprint planning discussion.

[tcraghu]

Catalog Draft Available at https://github.com/department-of-veterans-affairs/abd-vro-internal/wiki/Tokens-and-Secrets-for-VRO

[msnwatson]

As far as secret rotation goes, @yoomlam has completed mocks for the BIE Kafka which allow for use of self-signed certs and rotated secrets in dev. I have reached out to BIP (claims and evidence) teams for guidance on how to complete secret rotation but have not heard back yet. BIP secret rotation are the last pending item for this work before calling the meeting.

[msnwatson]

Going through the language in this ticket again with the additional context from working on it, I think we have most of AC1 done, with the exceptions of:

• documenting rotation/renewal instructions and expiry dates which we are working with @pshahVA to perform but which will likely be a longer process
• "how they [secrets] got there", this would be a very difficult task as written because of some high variance in the staffing of the team over time. I'm wondering if there is a way we can get more at the intent of this ask. Is this motivated by wanting to make sure that a secret had never been in source control history in the past. I think we can complete such a scan but accounting for the provenance of all secrets given a lack of consistent interaction with the project from team members might be difficult @va-albers @dianagriffin

Apart from these things, a catalog of the secrets as they stand is available at https://github.com/department-of-veterans-affairs/abd-vro-internal/wiki/VRO-Secrets

Also I think that we have kicked off part of AC2 and we are awaiting some more information from within the VA to either approve or request changes to some of our current methods in accounting for local development.

I have documented the current actions we are either waiting for answers or clarification for on the same page above in the hopes that we can rescope this ticket into a bit more manageable chunks.