nerc-project / operations

Issues related to the operation of the NERC OpenShift environment
1 stars 0 forks source link

Disable automatic networks blocks on MIT network(s) #738

Open larsks opened 1 week ago

larsks commented 1 week ago

I have lost access to systems hosted on the ESI external network (128.31.20.0/22) due to what looks like some sort of automatic policy implementation. This looks like a repeat from May 2024, in which we opened INC1327837 with MIT support (the incident is in theory available here), but that requires an MIT kerberos login).

There response at that time was:

You tripped a protective firewall configuration via using SSH repeatedly. This is often an issue with customers who are using git over ssh or sshfs.

For MIT community members, our standing advice is to get on the MIT VPN before initiating the connections; it's possible that there's client configurations that reduce the number of connections created, however.

I've unblocked your address; I'm going to pass the ticket over to the security team for their review.

This is problematic on a number of fronts:

We need to have the automatic blocking behavior disabled on these networks.

larsks commented 1 week ago

I have emailed servicedesk@mit.edu to attempt to get myself unblocked.

msdisme commented 1 week ago

While we resolve the broader issue, can we explore using a fixed IP for the demo and having them pre-approve it?

larsks commented 1 week ago

I would ask them about the broader issue first. In any case, I think it's unlikely that our demo will run into problems; the issue seems to be primarily stem from patterns of ssh access.

larsks commented 1 week ago

MIT has opened ticket INC1405264 for this request.

msdisme commented 1 week ago

discuss with csail champion first. email MIT asking about possible options to remove or loosen rules since many of users are outside of MIT

larsks commented 1 week ago

MIT has unblocked my home ip:

We have unquarantined your IP. It was quarantined for the same reason as last time (rapid fire SSH connections). For your reference, it was due to connections all to the same IP within a short window of time, over 45 SSH connections to 128.31.20.138 within three minutes (between 2:15pm EST - 2:18pm EST).

If you haven't already, we recommend setting up SSH ControlMaster configuration for this host.