CVMFS_KEYS_DIR setting too permissive

[DrDaveD]

Setting CVMFS_KEYS_DIR=/etc/cvmfs/keys will allow signing by any key under that directory, not just an individual key. That should either be changed to a subdirectory with just the galaxyproject.org key, or be changed to CVMFS_PUBLIC_KEY pointing to an individual file.

[DrDaveD]

In addition, the usual practice with CVMFS is to have a shared key per domain, not to have an individual key for every repository. The cvmfs_server tool doesn't do that automatically for you (except with the Yubikey signing feature), but it's easy to script.

[almahmoud]

This seems like a configuration choice, not an issue with the chart itself, given that the chart exposes the config files in values specifically for the purpose of allowing users to configure it as they desire. For the time being, this setup works out-of-the-box as expected for any of our repositories, and I don't see a benefit to changing it right now unless @natefoo thinks it's a major security concern.

[DrDaveD]

The CVMFS_KEYS_DIR setting is a significant security concern.

[natefoo]

@almahmoud I've taken to putting client keys under a galaxyproject.org subdirectory, although I wasn't aware of the security concern, it was more for organizational purposes. @DrDaveD is right, if any key in that directory can be considered valid for any configured repo, that's an issue.

Also,

In addition, the usual practice with CVMFS is to have a shared key per domain, not to have an individual key for every repository.

I only learned this last year well after our repos were in production and haven't yet looked in to whether there'd be a painless way to transition to a single key. One issue is that we don't control the majority of clients mounting our repos.

[natefoo]

@DrDaveD if I understand correctly, the issue with CVMFS_KEYS_DIR=/etc/cvmfs/keys would be that any key present in /etc/cvmfs/keys would be considered for any repo that's mounted? So I could create a (for example) foo.cern.ch repo, sign it with the data.galaxyproject.org private key, and get a client with the data.galaxyproject.org public key to mount it? Would the client still need a config for the rogue repo's upstream (excluding user mounts, which have their own configs) to exploit this?

@almahmoud is going to go ahead and move the keys to the subdirectory, we just want to make sure we understand the risk.

[DrDaveD]

Yes, in answer to your first two questions. For the third question, since all the transfers are through http (no https) and go through lots of public caches it's really not very difficult for an attacker to redirect traffic, so a client wouldn't need to also be configured to read from the attacker's server.

[almahmoud]

Changed in: https://github.com/CloudVE/galaxy-cvmfs-csi-helm/commit/e0528da5912663b687ff1833fe8ed0879bdd32a6 One last clarification, this is only a problem if someone has access to our repository private keys to sign their malicious repos, right?

[DrDaveD]

No, there wouldn't be a problem if that were the case. The problem is that anybody who has access to any private key corresponding to any public key in /etc/cvmfs/keys can pretend to be your repository.

[almahmoud]

There can't be any other keys in the container if not inserted through configmaps though, no? I'll further modify the image to come without the subdirectories for other repositories (cern.ch and egi.eu) cause there is no reason to ship with those subdirectories if we want k8s configs to be single source of truth, but there shouldn't be an issue if there's no other key in /etc/cvmfs/keys right? I'll still keep the subdirectory, just trying to understand if i'm missing something. How would keys be injected?

[DrDaveD]

There can be other keys inserted by rpms.

In any case, configuring a domain with its own keys is the right thing to do. Don't let all keys sign any domain.

[almahmoud]

I agree it's a better model. However, how can they be inserted by RPMs in a running container or a pre-built image, without access to the cluster? Also, why can they be inserted in /etc/cvmfs/keys and not /etc/cvmfs/keys/galaxyproject.org? Wouldn't it be the same risk if public keys got inserted there and there were malicious repos signed by the private portion of those keys, regardless of the subdirectory?

[DrDaveD]

I'm talking about keys that were inserted by rpms in the pre-built image. I'm not worried about those that are inserted maliciously later.

[almahmoud]

But there are none in the image that we're using. So it's only a problem if someone changes the image and uses a malicious image? And if inserted in a pre-built image, they can be inserted in the subdirectory too, right?

Sorry for pressing, I'm just trying to understand if this can actually be a security issue for an attacker to exploit, or if it can only be a problem if someone changes things wrong before deploying. In the first case, I would want to figure out what to do for the chart/image defaults to be secure. In the second case, I would argue it's a configuration problem on the side of the person deploying the chart/building the new CSI image. In that case, as long as our default configuration and our default image are not altered, it's theoretically secure? I am trying to clarify because we will be using this setup in an environment that HAS to be secure, and I don't want to overlook anything.

[DrDaveD]

If you have installed cvmfs, it requires an rpm that Provides cvmfs-config, so most likely you do have the cvmfs-config-default rpm installed, including the public keys for cern.ch, egi.eu, and opensciencegrid.org, unless you explicitly did something to prevent it. I'm quite sure that the owners of those keys are not going to do anything malicious with spoofing your repositories, but they could do it with the current config.