search

department-of-veterans-affairs / va.gov-team

Public resources for building on and in support of VA.gov. Visit complete Knowledge Hub:
https://depo-platform-documentation.scrollhelp.site/index.html
283 stars 204 forks source link

AWS access for Noah Stern #47401

Closed nfstern02 closed 2 years ago

nfstern02 commented 2 years ago

COR Name

Other - please specify in 'Additional Notes'

Vendor Onboarding Representative Name

GovernmentCIO - Kimberly O. West

Your Name

Noah Stern

Your Email

noah.stern@va.gov

Team, Role, and Company of the target individual

Developer VFEP (Education), GovCio/DBItPro

Product Manager (PM) name and email

Darla van Nieukerk - darla.vannieukerk@va.gov

Product Owner (PO) name and email

Michael Napper - michael.napper@va.gov

Desired AWS Access

I need access ssh access to EC2 staging, prod EC2 instances to execute the EDU GIDS Institution Builder and Geocoder process : We run the geocoder script.

To do so, you will need to find the instance id of staging-gids or prod-gids in the EC2 Instance search that will show a list of instances, select an instance and ssh.

• Once sshed into instance run “sudo su - -c 'docker exec -it gi-bill-data-service bash'” to bash into the container, “source config/.env.sh” to set the config enviornments. “nohup rake fix_coord_mismatch & exit” tpo start the geocoding task and sudo su - -c 'docker exec -it gi-bill-data-service tail -f nohup.out' to monitor.

Access Expiration

12/22/2025

Additional Notes

COR Name: Joseph Riccio - Joseph.Riccio@va.gov

Access to the production console is needed until I/we can figure out a way to automate the process more thoroughly. At the current time, we just inherited the process from another contracting firm that is rolling off. We need to study it and develop a deep understanding so that we can come up with a way to automate it. Until we can thoroughly automate it, we need access to the production console.

User must exist in a roster before AWS access can be granted

jbritt1 commented 2 years ago

Found in VFS team roster

jbritt1 commented 2 years ago

For AWS Access

User: Noah Stern COR: Joseph Riccio - Joseph.Riccio@va.gov VOR: GovernmentCIO - Kimberly O. West Approval needed: @department-of-veterans-affairs/operations-aws-approvers Admin level: no, but please note the request for production access

td-usds commented 2 years ago

Reached out to VOR

The request for Staging is fine with me, it's staging after all but... I have concerns about the requested level of access: it requests access to prod for too long a time and performs administrative operations (sudo) on instances in that environment. This is a serious risk.

Can you elaborate why this requires a human?

I would like to - at the very least - have a discussion as to what can be done to either limit the scope of the request and what the actual plan of action is to remove the need for a human in this loop that spawned this request.

nfstern02 commented 2 years ago

This is how the current process works which was developed by the current subcontractor AFS who developed the script. They are rolling off effective 9/29/2022 and we need to be able to perform the task as it stands until such time as it can be further automated.

nfstern02 commented 2 years ago

Maybe a compromise would be for the our team to gain access to prod for a 1 year to give enough time to continue prod support, and improve the situation?

td-usds commented 2 years ago

Can you give me some guarantees about the "until such time as it can be further automated" part? After all, a year is a long time for things to ... go "in different directions" or be re-prioritized into oblivion. I think that from a security perspective, we need some guarantees w.r.t. that automation part.

I think what I would like to see is a concrete plan and reasonable, measurable time-frame in which this automation would be completed. I understand that this is an inherited situation and this inheritance has highlighted an unfortunate security situation which I do think needs to be fixed.

If you want, this may be something where your PO needs to be involved in as it involves prioritized and I'm happy to have that discussion with them.

To be clear: if/when I do see that plan with a guarantee in terms of prioritization within a reasonable time-frame - because there is a serious security aspect to this - then I can sign off on this access request (with a hard termination timestamp of that reasonable date attached to the implementation plan). It is not inherently the access itself that I am concerned about(*) because I understand your predicament, it is the complete open-ended-ness of the highly privileged access and not having any enforcement mechanism to eliminate that open-ended-ness that I have a security issue with.

(*) To be clear: the highly privileged access is what I picked up on, but... semantics :)

nfstern02 commented 2 years ago

Based on conversations with AFS, they said they apply for access for 3 months at a time. Accordingly, revised the date to 12/22/2022. The plan is to automate this, but please understand this is our first exposure to the process and we are at the very front end of learning the nuances so we can automate it.

Thx, Noah

nfstern02 commented 2 years ago

We worked over the weekend to ramp up on what the script does and we think we have a strategy to automate the process so that we can eventually get out of the business of needing to manually kick this off from the prod console. Can you please revisit this ticket and grant access until 12/22/2022?

Thx, Noah

td-usds commented 2 years ago

@nfstern02 cf Slack message

td-usds commented 2 years ago

Reached out to VOR

markkvba commented 2 years ago

We'll be working this issue created by prior team to along to eliminate the need for PROD access: https://github.com/department-of-veterans-affairs/va.gov-team/issues/37944

td-usds commented 2 years ago

@markkvba confirmed that issue #37944 is being tackled with high priority and is the first issue they will be working on. Access for @nfstern02 approved for 2 months; based on the conversation with Mark, this is sufficient time for now.

jbritt1 commented 2 years ago

Just to be clear, user is approved for general access and 2 months of prod access. Is that correct?

mydesignrocks commented 2 years ago

@jbritt1 looks like @td-usds approved it for a temporary 2 month period. If so, is the process still the same with onboarding a new user to AWS? or anything different and if it is the same, which group does this user fall under?

jbritt1 commented 2 years ago

It honestly isn't clear to me which part / what is approved for the user....

If approved for general access, I personally would give Noah the groups previously used by the AFS team like Jacob Finnern here: https://github.com/department-of-veterans-affairs/devops/blob/master/terraform/environments/global/iam_users.tf#L704-L713. Noah and team aren't AFS I don't believe, so a new group will likely need to be created and named more appropriately by whichever team now owns the Terraform + IAM configurations given the recent shifts and platform reorg.

If prod access is also approved, I would add the adhoc-vetsgov-ssm-prod IAM group and set a Slack reminder for 2 months out to remove it. However, I am still not clear on exactly which part(s) have been approved for this user.

mydesignrocks commented 2 years ago

me neither, I didnt quite understand on whats approved and whats not for this user

nfstern02 commented 2 years ago

There was a guy from AFS named Patrick Arthur. I need to have access similar to his at least with respect to the instances that pertain to EDU GIDS Institution Builder and Geocoder process. Prod 2 months, staging open ended?

nfstern02 commented 2 years ago

AFS was the prior subcontractor that rolled off, so there's nobody on my team that can be used as a model to clone.

markkvba commented 2 years ago

It honestly isn't clear to me which part / what is approved for the user....

If approved for general access, I personally would give Noah the groups previously used by the AFS team like Jacob Finnern here: https://github.com/department-of-veterans-affairs/devops/blob/master/terraform/environments/global/iam_users.tf#L704-L713. Noah and team aren't AFS I don't believe, so a new group will likely need to be created and named more appropriately by whichever team now owns the Terraform + IAM configurations given the recent shifts and platform reorg.

If prod access is also approved, I would add the adhoc-vetsgov-ssm-prod IAM group and set a Slack reminder for 2 months out to remove it. However, I am still not clear on exactly which part(s) have been approved for this user.

Jeremy, What you propose maybe the most expedient way to get this done. From the GovCIO team we do not have any details provided by the prior team regarding which IAM groups they had. But the AFS Developer "Patrick Arthur" had the required access (staging, and PROD). The ec2 instances which we require access to "instance ids of staging-gids or prod-gids in the EC2" is the only detail we have been provided. Appreciate the assistance as this is becoming a blocker for the EDU business to update the School data.

If the needed IAM group is afs-vfs-developers, this groups should cloned/renamed into govcio-vfep-developers and given the same access policies?

jbritt1 commented 2 years ago

It honestly isn't clear to me which part / what is approved for the user.... If approved for general access, I personally would give Noah the groups previously used by the AFS team like Jacob Finnern here: https://github.com/department-of-veterans-affairs/devops/blob/master/terraform/environments/global/iam_users.tf#L704-L713. Noah and team aren't AFS I don't believe, so a new group will likely need to be created and named more appropriately by whichever team now owns the Terraform + IAM configurations given the recent shifts and platform reorg. If prod access is also approved, I would add the adhoc-vetsgov-ssm-prod IAM group and set a Slack reminder for 2 months out to remove it. However, I am still not clear on exactly which part(s) have been approved for this user.

Jeremy, What you propose maybe the most expedient way to get this done. From the GovCIO team we do not have any details provided by the prior team regarding which IAM groups they had. But the AFS Developer "Patrick Arthur" had the required access (staging, and PROD). The ec2 instances which we require access to "instance ids of staging-gids or prod-gids in the EC2" is the only detail we have been provided. Appreciate the assistance as this is becoming a blocker for the EDU business to update the School data.

If the needed IAM group is afs-vfs-developers, this groups should cloned/renamed into govcio-vfep-developers and given the same access policies?

I am game to give you and the team anything you need, we just need clarification on which part(s) @td-usds is approving.

As far as cloning the IAM group, that is the logical approach I would take it if we’re up to me. However, as of today I am operating in a reduced capacity on this project and ramping up on another for the next several weeks so I will likely not be the one owning this task.

markkvba commented 2 years ago

The only issue in question was the prod access/duaration. Staging, devl etc was not at issue. @td-usds approval, therefore means to us, a non-expiring staging access, and 2 month prod access to the ec2 instances needed for gids.

jbritt1 commented 2 years ago

The only issue in question was the prod access/duaration. Staging, devl etc was not at issue. @td-usds approval, therefore means to us, a non-expiring staging access, and 2 month prod access to the ec2 instances needed for gids.

FWIW, that’s my interpretation as well. However, I wanted it to be clear as platform security and access management has been a topic of increased scrutiny for a while now. I do understand and sympathize with the frustration from the delay, it is not my intent to add to that.

markkvba commented 2 years ago

Understood, thank you.

jbritt1 commented 2 years ago

@markkvba, I did a lot of thinking on this last night and am going to move forward with general access for Noah right now. I would have liked to have gotten clarification, but am making a judgement call on this one. If the interpretation winds up not being correct, I will take the hit on the misunderstanding of it. I will reach out to Noah in a bit with some first-time login credentials.

mydesignrocks commented 2 years ago

@markkvba, I did a lot of thinking on this last night and am going to move forward with general access for Noah right now. I would have liked to have gotten clarification, but am making a judgement call on this one. If the interpretation winds up not being correct, I will take the hit on the misunderstanding of it. I will reach out to Noah in a bit with some first-time login credentials.

thanks @jbritt1 - on that note, we should catch up some time on role / group definition (aka separation) as and when we all settle in our new roles.

jbritt1 commented 2 years ago

@markkvba, I did a lot of thinking on this last night and am going to move forward with general access for Noah right now. I would have liked to have gotten clarification, but am making a judgement call on this one. If the interpretation winds up not being correct, I will take the hit on the misunderstanding of it. I will reach out to Noah in a bit with some first-time login credentials.

thanks @jbritt1 - on that note, we should catch up some time on role / group definition (aka separation) as and when we all settle in our new roles.

Agreed, but would have to point to @ph-One (Kyle Matheny) or @little-oddball (Clint Little) on that one as I am actually supposed to be working full-time on another project for the next several weeks and will have limited bandwidth to bounce between that project and this one.

jbritt1 commented 2 years ago

@markkvba, I have a pull request up to add Noah's user and will get that applied and merged upon approval from a codeowner. Once that part is done I will pass him his first-time login credentials.

jbritt1 commented 2 years ago

First-time login credentials passed to @nfstern02 via DSVA Slack workspace DM at this time. Leaving the ticket open however until we get clarification on the prod access part.

nfstern02 commented 2 years ago

We got a request from the user to run the script in prod and I got this error when I tried. (data x'd out for security reasons)

User: arn:aws-us-gov:iam::xxxxxxxxx:user/Noah.Stern is not authorized to perform: ssm:StartSession on resource: arn:aws-us-gov:ec2:us-gov-west-1:xxxxxxx:instance/i-xxxxxxxxx because no identity-based policy allows the ssm:StartSession action

I received an email dated 10/3 from Thomas D. stating:

Access for @nfstern02 approved for 2 months; based on the conversation with Mark, this is sufficient time for now.

We are now officially blocked from supporting our user. Please assist.

markkvba commented 2 years ago

First-time login credentials passed to @nfstern02 via DSVA Slack workspace DM at this time. Leaving the ticket open however until we get clarification on the prod access part.

The following event indicates the approval was granted (since the issue applies to both staging and prod):

@td-usds td-usds removed the Awaiting Approval need approval before processing label 11 days ago

td-usds commented 2 years ago

w.r.t. https://github.com/department-of-veterans-affairs/va.gov-team/issues/47401#issuecomment-1265693800

@markkvba confirmed that issue #37944 is being tackled with high priority and is the first issue they will be working on. Access for @nfstern02 approved for 2 months; based on the conversation with Mark, this is sufficient time for now.

Apologies, I did intend to mean this to be 2 months granted for the requested access including prod...

nfstern02 commented 2 years ago

Can you state that prod access should be granted?

jbritt1 commented 2 years ago

Adding prod perms now for a 2 month period. Will set a reminder for the DevOps CoP to remove in 2 months as I will likely still be on the other project I have been assigned to or paternity leave.

jbritt1 commented 2 years ago

Reminder set for 2 months out in the platform-devops channel

image