[Network/Security] CSOC needs connectivity to conduct Annual Penetration Testing

troymosher commented 2 years ago

Description

The following dedicated PenTest AWS VAEC test systems require connectivity (ping and network access to all open ports) to Platform systems/IPs in order to conduct Annual Penetration test:

10.247.2.43 10.247.2.37

Background/context

Annual Penetration testing is an ATO requirement and is due by 3/31. We requested CSOC conduct an annual pen test of our system, and they replied that while they are able to see the systems are online, they are unable to conduct the pen test as their dedicated systems do not have the ability to ping or access to all open ports for our environment. They need this in order to conduct the Pen test.

Technical notes

According to Boris, this may require configuring both the TIC firewall and AWS security groups.

Tasks

[ ] Allow two systems listed above to access all ports for Platform systems.

Acceptance Criteria

[ ] The dedicated PenTest systems are able to ping and connect to all open ports for Platform systems.

Reminders

[ ] Please attach your team label and any other appropriate label(s) (operations, devops, and needs-grooming will automatically be applied as part of the template)
[ ] Please connect to an epic (this will typically be done by the Platform Operations PM or TL)

troymosher commented 2 years ago

@jhouse-solvd I know infra has EKS as it's priority, but with this task being time sensitive with the pen test being due 3/31, do you think it will be accomplished within the next 6 weeks?

cc: @td-usds

jhouse-solvd commented 2 years ago

@troymosher - Thanks for the mention. I'll add this to our list of To dos, and discuss with the team this sprint. It seems straightforward, though may require ESECC requests to open ports at the TIC (I believe?).

jhouse-solvd commented 2 years ago

@troymosher - We discussed as a team and believe that these actions could be performed by the Security team's DevOps engineer. That engineer should have the access required to do it, and if not, we can grant the necessary permissions.

Please let us know your thoughts.

td-usds commented 2 years ago

It's less about access and more about separation of duties, I think. We certainly could do the thing, but we're also part of the group of folks that should/will be evaluating it independently so we shouldn't be the ones doing the thing. (fox, hen house, and all those things) This request has a due date of 2022-03-30 (the pen test is scheduled on 2022-03-31), I think that's far enough ahead in the future for it to enable it to be scheduled appropriately. This is not a "do this now", this is a "we need this in 40 days/5.5 weeks".

mchelen-gov commented 2 years ago

@td-usds @troymosher Can you clarify this request:

What is meant by "all platform systems"
What specific ports are being requested

For awareness Infrastructure team has no ability to alter TIC configuration.

troymosher commented 2 years ago

@jhouse-solvd CSOC is requesting connectivity to ping and access all open ports on our systems from their two dedicated pentest devices (10.247.2.43,10.247.2.37). They would presumably need all ports open to all devices within the 10 VPC's in our boundary. It's presumed they have access through the TIC, but still need to be allowed in our AWS security groups.

jhouse-solvd commented 2 years ago

This would probably require:

create a security group that opens access from their CIDR range to platform instances/services on desired ports
ensure new security group is applied to all relevant machines

One potential limitation is the number of security groups that can be attached to an instance.

dginther commented 2 years ago

It's less about access and more about separation of duties, I think. We certainly could do the thing, but we're also part of the group of folks that should/will be evaluating it independently so we shouldn't be the ones doing the thing. (fox, hen house, and all those things) This request has a due date of 2022-03-30 (the pen test is scheduled on 2022-03-31), I think that's far enough ahead in the future for it to enable it to be scheduled appropriately. This is not a "do this now", this is a "we need this in 40 days/5.5 weeks".

I don't think separation of duties really applies here, in my opinion. This is not a change that will be audited or evaluated by the Security team, it is a change that is necessary for the security team to get pen test results, and therefore seems to be something that could be performed by the security team to enable the pen tests to get done in a timely fashion, since the infra team is overburdened at the moment. Just my $.02.

Also in my opinion, this is a terrible request for the pen test team to be making. An attacker would not be whitelisted on all of the devices in the network that they were attacking. This is an example of the 'penetration test' not being an actual penetration test, but rather a scan with Nessus or some similar tool, and a regurgitation of whatever that tool found, perhaps with some proofs of concept to show how someone might exploit any vulns that were found. A penetration test should be performed with the system as it is found, without extra whitelisting, etc.

If Nessus or other security tool scans should be performed from internal resources to discover vulnerabilities, the security team should bring up a scanner inside the VPC to be scanned and scan these things themselves, and provide the output of that scan for eMASS.

jhouse-solvd commented 2 years ago

This looks like an issue where something similar was done previously #27482

joeniquette commented 2 years ago

Speaking from both VSP Eng lead and Infra lead here are my recommendations:

The security team should be able to provide the list of ports they want opened and to what IP addresses. The infrastructure team makes the appropriate changes to the security groups through terraform. After the test is complete the infra team reverts the change.

I'm not a fan of teams making security group changes to services that they dont manage/use day to day. The security team doesn't directly interact with any of these services (yet). If say the security team deployed a security solution that they managed every day, then yes they should put the PR in with the terraform code change to update the security group. They however dont manage the infrastructure day to day, the infra team does. So for now, until things change, we should be making the appropriate changes provided the security team clearly dictates the requirements. If the infra team doesnt know how to best make these changes we can get some folks to consult with the infra team (probably myself and someone from the security team).

@td-usds and @troymosher Can you please send over a spreadsheet with the "to and from" IP addresses and ports you need updates be made for. Include duration of change and contact (pen test team) to test if the change is successful? I expect this to only be a change to our security groups in AWS.

@jhouse-solvd Once we have the spreadsheet can we please make a PR to update the appropriate security groups for the appropriate VPCs?

troymosher commented 2 years ago

@joeniquette CSOC has two dedicated "pentest" devices 10.247.2.43 and 10.247.2.37 that need access to all open ports on all the devices within our authorization boundary. (ie. our 10 VPCs)

The "pentest" is expected to last 30 days according to the SOP, and any deficiencies would need to be mitigated and then re-tested, so it could conceivably need to be in place for quite some time, until we get a clean bill result.

Also, as this is an annual requirement, and the two dedicated CSOC devices are expected to be static over time, a solution where we could build this into terraform and turn on and off as needed annually would be ideal.

We have an open request ticket on the VA side for the pentest and the CSOC team is standing by to test if the change is successful.

dginther commented 2 years ago

Why did we bother putting a devops person on the security team?

Why did we bother splitting all the teams up into cross-functional teams, if each team just now has a single person who is the 'conduit' for requests to the other teams?

The idea here would be to have the devops person on each cross functional team performing the functions that you would normally ask the 'ops' team to do, or at least that was the rationale that I understood from the talks about how to separate and make cross functional teams.

By having each 'devops' person on the individual teams make the PR to do the work, you accomplish multiple things:

The person on each team gets more familiar with the platform and what is needed to make changes
Load on the infra team is reduced so they can perform the actual infrastructure work
PR review process is used to increase the number of eyes on the changes being made, and to increase visibility within the organization of the changes being made

Saying that the security team is not 'using' these services is, in my opinion, irrelevant. The security team is not a 'user' of any services on the platform, except the ones that they might create, but that doesn't excuse them from being responsible for the security of the platform as a whole.

The model you describe for this is untenable already. The infra team was 9 people, and swamped with requests from all the other teams, and unable to perform their daily infrastructure work because of those interruptions. They are now 6 people, and have lost much of the institutional knowledge they once had due to burnout from those constant requests and interruptions. The cross-functional team idea was supposed to help this situation by pushing those requests and the fulfillment of those requests to the requesting team themselves, where the 'ops' person on each team would take care of the request, so that the infrastructure team could focus on the infrastructure.

dginther commented 2 years ago

@joeniquette CSOC has two dedicated "pentest" devices 10.247.2.43 and 10.247.2.37 that need access to all open ports on all the devices within our authorization boundary. (ie. our 10 VPCs)

The pen test is not limited to the 'staging' VPC? You are going to allow scans and testing on the prod VPC?

Also, as this is an annual requirement, and the two dedicated CSOC devices are expected to be static over time, a solution where we could build this into terraform and turn on and off as needed annually would be ideal.

I renew my objection to the idea of this. A valid penetration test is one which tests the system as it sits. This is not a penetration test, this is a vulnerability scan. That's all well and good, however, it does not give us any actual data about our threat profile or level of risk, as it tests from the perspective of an inside threat, not an outside threat.

td-usds commented 2 years ago

The security team will do the legwork on this ticket; apologies to the infra team for the randomization on this one.

td-usds commented 2 years ago

@azeez-salu, can you take this one?

dginther commented 2 years ago

@azeez-salu If you need any assistance on this, or aren't quite sure where to get started, I am happy to give you a hand in the right direction.

ffafara-tw commented 2 years ago

Please include VANotify team. VANotify is a minor system under VSP ATO but is in a separate AWS account. @nathanbwright @bevnobev @Eallan919

azeez-salu commented 2 years ago

@td-usds I had a meeting with Gary, and I have good information on what I need to do.

Thanks @dginther. I will reach out when I have questions.

department-of-veterans-affairs / va.gov-team