search

department-of-veterans-affairs / va.gov-team

Public resources for building on and in support of VA.gov. Visit complete Knowledge Hub:
https://depo-platform-documentation.scrollhelp.site/index.html
281 stars 197 forks source link

request delete of old MVI/MPI PS objects #69275

Open joeniquette opened 10 months ago

joeniquette commented 10 months ago

The following should be removed from PS related to the most recent MPI Cert changes. We were able to do this because we consolidated the references to the certs to fewer values. We also reduced our lower environment certs to a single cert (down from 4). Finally we removed the PS value pointers to cert paths that can just be hardcoded paths inside the config files. We did not change anything for production yet but we will in the next ticket.

Already Removed (Identity team has permission in AWS):

  1. /dsva-vagov/identity-team/fwdproxy/int/mvi_key
  2. /dsva-vagov/identity-team/fwdproxy/preprod/mvi_key
  3. /dsva-vagov/identity-team/fwdproxy/qa/mvi_key
  4. /dsva-vagov/identity-team/lowers/mvi_qa_cert_dev.pem
  5. /dsva-vagov/identity-team/lowers/vetsgov-mvi-int-cert.pem
  6. /dsva-vagov/identity-team/lowers/vetsgov-mvi-preprod-cert.pem
joeniquette commented 10 months ago

I also retired the int, preprod, and pint certs in venafi for this connection.

ph-One commented 9 months ago

@barbarello may be interested in this work

barbarello commented 9 months ago

looking into this

barbarello commented 9 months ago

Manually deleted 6 (managed) via terraform and 21 (unmansged) via aws console.

barbarello commented 9 months ago

turns out #20 (/dsva-vagov/vets-api/sandbox/mvi/url) should not haven listed in the first place as it is being used in https://github.com/department-of-veterans-affairs/vsp-infra-application-manifests/

rmtolmach commented 9 months ago

@joeniquette @barbarello /dsva-vagov/vets-api/sandbox/mvi/url was deleted from AWS, but it wasn't deleted from the manifest repo (maybe it was supposed to be done in this PR? https://github.com/department-of-veterans-affairs/vsp-infra-application-manifests/pull/2368). We discovered this when vets-api sandbox wouldn't deploy yesterday 😟. @LindseySaari added it back to AWS so we could deploy. If it does need to be deleted, can you remove it from the manifest repo and then after the 3pm ET deploy, ping me or a Platform devops eng to delete it from AWS.

rmtolmach commented 9 months ago

jinx @barbarello 😆

barbarello commented 9 months ago

@rmtolmach it wouldn't make sense to delete it from parameter store if pods are using it.

rmtolmach commented 9 months ago

@barbarello the value would need to be deleted from here first and then deleted from Param store, if that's what Joe wants. Or we just leave it in both places. The scenario yesterday was that it was only removed from PS but it was still in the manifest.

barbarello commented 9 months ago

@rmtolmach agreed on the cleanup process. What i'm trying to communicate here is that if there remains an operational need for a key, which is from a pod's perspective, it should not be deleted from paramter store.
So , just saying keys are no longer needed is not enough.

rmtolmach commented 9 months ago

@barbarello got it. Looks like it is used https://github.com/department-of-veterans-affairs/vets-api/blob/master/lib/mpi/configuration.rb#L25

https://github.com/search?q=repo%3Adepartment-of-veterans-affairs%2Fvets-api%20MPI%3A%3AService.new&type=code

barbarello commented 9 months ago

unchecked items 1. through 8. as they are used in https://github.com/department-of-veterans-affairs/devops

joeniquette commented 9 months ago

@barbarello The sandbox mvi url was likely a miss-paste sorry about that. Good to know its actually being used, even if deleting it was the validation something used it.

As for 1-8, those values have not been updated with the new cert values, and all the systems are working even though the old certs have now expired. We also consolidated all the certs in lowers to one, its the qa one. So anything that points to an "int", "pint", or "preprod" no longer has a valid cert. As for the others, the only places I could find that uses them are in the staging worker and server api configs. Which the file says its archived and to not make updates. The PS values in this list:

  1. /devops/certificates/mvi_int_cert_dev.pem
    1. /devops/certificates/mvi_int_dev.key
    2. /devops/certificates/mvi_qa_cert_dev.pem
    3. /devops/certificates/mvi_qa_dev.key
    4. /dsva-vagov/fwdproxy/int/mvi_key
    5. /dsva-vagov/fwdproxy/pint/mvi_key
    6. /dsva-vagov/fwdproxy/preprod/mvi_key
    7. /dsva-vagov/fwdproxy/qa/mvi_key

did not get updates to new cert values which I think validates they aren't needed since we have working systems. I am in favor of validating this theory, but if proved true, I do not want values in PS which contain the term "mvi" that are orphaned. They make cert changes problematic because then we have folks trying to figure out why they weren't updated, what should they do, etc.

joeniquette commented 7 months ago

@rmtolmach it looks like @barbarello isnt on the project team anymore. Is this something someone else can finish?

rmtolmach commented 7 months ago

👋 Passing along to the DevOps CoP lead @pjhill

jwoodman5 commented 6 months ago

@pjhill to add to DevOps CoP meeting agenda 2/15.

pjhill commented 6 months ago

@stoiven -- will address this as he has time during the current support rotation and hand off any remaining items when the next devops support person comes on

pjhill commented 6 months ago

Per Joel -- beware possibility of accidentally removing items that are actually still in use.

We need additional clarity around how to coordinate this effort before we can work this more.

joeniquette commented 5 months ago

@pjhill any updates on this ticket?

pjhill commented 4 months ago

No updates in a while -- I'll need to dig up an appropriate summary of recent activity.

joeniquette commented 3 months ago

@pjhill just doing my monthly check-in on this ticket, any updates?

pjhill commented 2 months ago

Parameter store and certificate contained within it are being cleaned up as a low priority item as members of DevOps COP scattered across various Platform teams come into contact with aspects of it. I will bring up this list in the DevOps COP meeting this Thursday -- 6/27.