thoughts about a possible security exploit

MatthiasWinzeler commented 6 years ago

hi all

we're thinking about offering nfs volumes in our CF. however, we're very concerned about the security:

on our NFS server, we'd have to open the exports to all the Diego cells, since it's NATted (as described in the README
thus, every app or cell could access all NFS shares of all users
to mitigate this, we would disallow the apps from accessing the NFS server's IP by using security groups

but: it looks to me that nfs volume driver mounts all service bindings that follow a specific format (volume_mounts). what if a user pushes his own service broker that gives out service bindings with volume_mounts that points to a victim's NFS share IP? would then the driver happily mount the NFS share?

in nfsv3driver can I somehow whitelist/blacklist brokers so that only 'our' broker is allowed to give out nfs mount points? or can we somehow lock down the driver so that above attack vector would be impossible?

p.s. using ldap is not an option, since the user could use above 'adversary-broker' to circumvent the ldap check by giving out a binding that doesn't contain the ldap stuff. this could probably be mitigated if we can force ldap on the nfsv3driver level, but as far as I can see, this is not possible atm.

cf-gitbot commented 6 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/158638881

The labels on this github issue will be updated when the story is started.

julian-hj commented 6 years ago

HI @MatthiasWinzeler.

It is possible to disable uid and gid settings in the nfsv3driver, in which case, ldap would be the only way that a user can resolve to a particular UID. Our thinking was that provided file access on your share is not open to all UIDs, that should give you a measure of protection.

Regarding the possibility to create a malicious service broker, the user must be an admin in order to register a service broker, and if they are an admin, then they can also manipulate security groups.

Cloud Foundry supports the create-user-provided-service for non-admin users to register services without a service broker, but we do not support volume mounts for user provided services, for precisely this reason.

Does that make sense?

MatthiasWinzeler commented 6 years ago

thanks for the answer, @julian-hj!

The option with uid and guid could indeed provide such a measure. how would we lock down a user's share with this mechanism, though? I assume as soon as ldap auth is completed, a user can access every share (also other user's shares)... so we'd need kind of an intelligent ldap server that only grants access if a user tries to mount his own share. And I guess the ldap server does not get the mount point a user is trying to access..?

About service brokers: I think there is the possibility for space developers to register space-scoped service brokers: https://docs.cloudfoundry.org/services/managing-service-brokers.html#register-a-broker

Created by space developers using the command cf create-service-broker with the --space-scoped flag

julian-hj commented 6 years ago

Regarding space scoped service brokers, that is a good point. I don't think we do anything to disallow space scoped brokers.

Regarding how LDAP works, the normal setup would be that the same LDAP server that is used to populate accounts onto the NFS server is also used by nfsv3driver to resolve usernames into UIDs. So, if a share is owned by a particular user on the nfs server, and only visible to that user, then it's not possible for other users to access it because they will resolve into different UIDs.

This is more or less how security works in NFS v3. The nfs server uses ip addresses to decide what other hosts it trusts, and then those hosts are responsible to send correct uids via nfs rpc.

The property you would use to disable uid and gid in a service binding is nfsv3driver.allowed-in-source. This defaults to allowing uid and gid, so you would want to set that to empty.

MatthiasWinzeler commented 6 years ago

@julian-hj thanks, that explains it to me. we'll check whether our nfs share supports proper ldap integration.

are there any other security features planned in the future? I saw some kerberos strings in the nfs driver, but it doesn't look like they're used.

julian-hj commented 6 years ago

We did some investigation of kerberos with nfs4 a while ago, with the idea that if we could enable kerberos authentication in nfs connections, we'd get much better security. While that is true in theory, we eventually abandoned it because: 1) kerberos is sort of the opposite of cloud native and getting the client software to interact with our volume drivers was difficult and error prone. 2) setting up kerberos authentication is quite difficult even without introducing cloud foundry into the picture, so when we interviewed customers about it, we didn't find any actually using kerberos, although there were some who felt like they ought to.

We support nfs4 in the latest versions of nfs-volume-release, but I think that will only buy you very modest security gains without kerberos enabled.

You might also consider using smb-volume-release if Samba shares are an option for you from your file server. Since CIFS/Samba is a single-user protocol that has user credentials built in, it provides a more robust mechanism for you to secure your shares against unauthorized mounts. Smb3 also supports encryption of data in flight, which we cannot do in NFS without kerberos turned on.

julian-hj commented 6 years ago

@zrob I think there is a case to be made here that space scoped service brokers requiring volume mounts should be rejected on security grounds. Would you be open to a PR to add that restriction? Since it would disallow something previously permitted, we'd definitely have to put it out to cf-dev first to make sure there are not use cases we haven't thought of, but I am not aware of anyone legitimately registering space scoped brokers that require volume mounts.

julian-hj commented 6 years ago

@MatthiasWinzeler we've been looking into this issue some more, and trying to decide if it makes sense to add another UAA scope to control which users can register volume service brokers.

In the course of that investigation @mariash noted that we already have a feature flag space_scoped_private_broker_creation (https://docs.cloudfoundry.org/adminguide/listing-feature-flags.html) that can be used to turn off the ability of space developers to register space scoped service brokers.

Do you think that disabling space_scoped_private_broker_creation would be a viable alternative for you, or do you have use cases that require your space developers to be able register other kinds of service brokers?

MatthiasWinzeler commented 6 years ago

@julian-hj thanks for following up! However, we indeed allow our customers to create space scoped service brokers and would still like to prevent them from creating volume service brokers.

julian-hj commented 6 years ago

@MatthiasWinzeler ok, thanks for the speedy response. We'll do some brainstorming around here to see if we can find a way to solve this with better granularity.

That could be: 1) disallowing any non-admin from registering volume service brokers at any scope 2) adding a feature flag like space_scoped_volume_service_broker_creation to act like the other feature flag, but only for volume services 3) adding a UAA scope that non-admins require in order to register a volume service broker

None of these options seems very appealing, but we will see what we can do.

julian-hj commented 5 years ago

closing due to inactivity

cloudfoundry / nfs-volume-release

thoughts about a possible security exploit #36