Audit Credstash against SSM Param Store

[oseasmoran73]

Description

The transition from credstash to AWS param store is partially complete see procedure. We need to know which keys remain in credstash that the Platform Infrastructure team (PIT) needs to migrate.

Background / Context

In summer 2021, a script was run to sync AWS param store with credstash. Link to the directory with script.

For any keys that the PIT is not responsible for, we can develop communications and documentation to share with other teams, with a view to completely sunset credstash usage later this year. (separate issues and initiative)

Tasks

• [ ] Document credstash audit workflow
• [ ] Document credstash keys currently in use by the Platform Infrastructure team
• [ ] Create issues to track the migration of credstash keys currently in use by the Platform Infrastructure team

Acceptance Criteria

• [ ] Documentation exists that allows us to audit credstash in the future easily
• [ ] Issues have been created for credstash keys that the PIT needs to migrate

[rmtolmach]

The transition is slowly happening when someone puts up a PR and changes certain files. See https://github.com/department-of-veterans-affairs/devops/pull/11125 as an example. There is a bot message that tells them what to do:

Please update the corresponding values in the k8s ConfigMap https://github.com/department-of-veterans-affairs/vsp-infra-application-manifests/blob/main/apps/vsp-tools-backend/vets-api/dev/settings-configmap.yaml. If you have updated a secret value, please also update the corresponding secrets.yaml https://github.com/department-of-veterans-affairs/vsp-infra-application-manifests/blob/main/apps/vsp-tools-backend/vets-api/dev/secrets.yaml. Additional context: https://vfs.atlassian.net/wiki/spaces/OT/pages/2146140164/Keep+EKS-Deployed+Vets-API+Up+to+Date+with+BRD-Deployed+Vets-API

I think we should document how to do this two-step process (1. make sure things were migrated correctly when the automatic 2. Remove all references to old credstash value and update with new param store value) so we're all on the same page of what the tasks are.

Ideas:

• Infra team members do 3 per day.
• We could organize a "fun" day where we all spend the day doing this.
• Person on-call for the week does a few every day, as time permits.
• We all spend a couple hours at the end of the sprint doing this.
• Matt does it all.
• Brainstorm ways to make this less manual and painful.

[jhouse-solvd]

To summarize, it would be good to know

• What's still in credstash
• Which of those things are the responsibility of the Platform Infrastructure team
• Which of those things are not

[jhouse-solvd]

@rmtolmach is going to get started on this next week.

[rmtolmach]

Number of credstash mentions in devops repo organized by directory: https://vfs.atlassian.net/wiki/spaces/~121859534/pages/2231402521/Credstash+keys

[jbritt1]

PLEASE NOTE: anything that is built / deployed in the commercial AWS account will need necessary Parameter Store values duplicated there in order to work.

[rmtolmach]

Moving back to the backlog. For reference, there is documentation written up on how to do this (linked in the description). See the How to update a value to be used from Parameter Store instead of Credstash section.