Specifying security type of mount

[briantopping]

It's unclear how to specify a mount option such as sec=sys. When using a NFSv4 volume from a modern operating system, the negotiation will naturally take the highest level of security that is available.

Unfortunately, if that resolves to krb5, an arbitrary container is unlikely to have a ticket or the ability to acquire one. In that case, the contents of the volume will either be inaccessible or mapped to nobody. sec=sys is not ideal, but sufficient in a controlled security context.

Apologies if this is already addressed and I missed it in the documentation.

[codenrhoden]

@briantopping You haven't missed anything so far, as the documentation is pretty thin.

I'll be refreshing this code quite a bit after the first of the year, as the CSI spec finally nears 0.1 status. As it stands today, it is quite a bit different from the current state of the spec, especially as it pertains to the Volume IDs.

The current version of csc available at thecodeteam/gocsi is geared toward the current spec of CSI, so you'd have to use an older version to talk to csi-nfs anyways. But if you did, the command to pass the mount option would look similar to csc node mnt -o sec=sys .... The -o flag is supposed to pass the mount options straight through, but at the point the plugin is today, I cannot attest to whether that works or not... :/

[briantopping]

Awesome, thanks for the detail, Travis!

I'm using the plugin from within REX-Ray, so csc and anything to do with how to configure a CSI plugin from within /etc/rexray/config.yml is still serious voodoo. I've read the sources for csi-nfs several times and have gotten nowhere. It's the excuse I've needed for a couple of years to learn Go, just gotta weave it in with the rest of life.

So I suppose what I can offer right now is "beginner mind" kind of feedback and end up meeting you halfway as I get better with Go and the rest of life allows you the opportunity to get the next release out.

Glad the answer wasn't RTFM! 8)

[codenrhoden]

@briantopping And thank you for the feedback. Fresh perspectives welcome!

We are definitely in early days with CSI, and the REX-Ray integration to the csi-nfs plugin is new as well. I highly suspect there is no passthrough of those mount options at this time, but it's great to know that (1) someone is trying and needs it, and (2) it probably does not work. :)

[briantopping]

Ok, so here's a thought experiment: If I were magically able to build a PR that was accepted for the the REX-Ray integration to the csi-nfs plugin, what would be the first file I would want to look at? I don't want to burn up your time, I might not be successful. But I think the answer to this will be very telling... :)

[briantopping]

So I've spent a couple of days tweaking the configuration to work around this issue but am coming up empty.

As of this morning, I was wondering why a volume was mounting just fine on alpine:latest but not with prom/prometheus:latest. As it turns out, this image has a directory at the mount point and the directory causes the volume mounter to copy the files from the image to the directory when it's mounted. The nocopy flag to docker-run sidesteps this copy and allows the container to launch, but there's no way to add the flag with Marathon, so I can't determine if that would solve my current issue. I tried a few other tweaks, including changing the directory on the NFS mount to nobody:nogroup/777 and 0:0/777 but both of these fail in docker with a debug message of copying image data. I was also able to reconfigure the server to only negotiate sec=sys for the particular mount, which is working according to nfsstat -m when successfully launched with nocopy. The behavior also does not change with the addition of --privileged, which is quite odd. I'll start trying to dig into the Docker source to discover what I can.

There's not quite enough debug data there to explore it beyond this message that there's something going wrong with the copy and it's weird that dockerd appears to be running as root (why would the copy fail), but it's also odd because the directory that it's copying from the container to the mount is empty.

Apologies for the TMI, just sharing what's going on in the field...

[codenrhoden]

Hi @briantopping,

I haven't tried running things under Marathon myself, so I'm afraid I'm not much help with what you are toying with. I am going to update REX-Ray's docs to have an example of using csi-nfs with Docker, but I doubt it will be much help today, because...

If I were magically able to build a PR that was accepted for the the REX-Ray integration to the csi-nfs plugin, what would be the first file I would want to look at?

You would want to look at: https://github.com/thecodeteam/rexray/blob/master/agent/csi/mod_csi_docker_bridge.go There's a lot of NFS specific code in that file. However, as for being able to modify/influence the mount options, it's a bit of a dead-end, actually. We are extremely limited by Docker's volume API here. When we mount the volume, we have to create a "volume capability", which is done here: https://github.com/thecodeteam/rexray/blob/bf500f31232f414f85486d2697cf4e6bfe1a59f0/agent/csi/mod_csi_docker_bridge.go#L881-L885

That is setting the filesystem type, and, is also setting the mount flags here: https://github.com/thecodeteam/rexray/blob/bf500f31232f414f85486d2697cf4e6bfe1a59f0/agent/csi/mod_csi_docker_bridge.go#L923-L937

If you look close, you'll notice that no mount flags are ever passed in. That is because Docker doesn't give them to us. The "mount" under Docker happens when you do a docker run, but there is no CLI option when doing docker run to pass in options to a volume driver. For example, you can't do something like docker run --volume-driver rexray --volume-opts=-o=sec=sys .... There is no way to do it. Docker forces you to define all "options" for a volume when you do a docker volume create, but even then, it doesn't pass any of those options along to the volume driver when it does a mount. The only information we have to work with is this: https://github.com/docker/go-plugins-helpers/blob/bd8c600f0cdd76c7a57ff6aa86bd2b423868c688/volume/api.go#L36-L40

Which is not very useful. REX-Ray tries very hard to be stateless today, which precludes the option of caching those options when doing a create and re-using them during a mount. That info would be lost on a restart of REX-Ray. There would also be no way to override those options when doing a mount (like doing -o ro on one node and -o rw on another...

I was going to link you to the docs showing how to use Docker's native NFS integration, but as I was looking for it, I found something new to me, which is the --mount flag now being available to single containers instead of only in Swarm mode. This would mean what I said above about not being able to pass in options to a volume driver is no longer true, which is great news. This is something I will be able to build off of in the future, but I don't have a timeline for you. Specifically, I was looking at: https://docs.docker.com/engine/admin/volumes/volumes/#create-a-volume-using-a-volume-driver

[briantopping]

Wow, manna from heaven! Thanks @codenrhoden! I'm going to burn on this through the weekend, this is a real gem of a message. Happy Friday to you!

[briantopping]

Hi @codenrhoden, thanks again for your work on these, including your recent doc on CSI-NFS in RR. I left a comment at https://github.com/thecodeteam/rexray/pull/1133/files#r157861451, I considered there that I may be misapplying RR and friends, expecting it to fill the gap that non-Swarm environments (or at least DC/OS) have WRT volume management across agent nodes.

If the only focus of RR is to act as a venue for storage and orchestrators to meet (laudable in itself, of course), then I'm left wondering if DC/OS is leaning on RR for too much and the problem is actually poor volume management in DC/OS. To your point in the PR above, CSI-NFS is not intended as a replacement for the local NFS services available in https://docs.docker.com/engine/reference/commandline/volume_create/#driver-specific-options. Nevertheless, a large DC/OS deployment still needs to have the volumes maintained across all agents and I may have mistakenly projected the gap from the DC/OS docs that RR configs made this possible.

So I guess my next direction ought to be whether cluster-wide volume management is scope creep for RR. If so, I need to re-analyze how I'm going to solve that first, then see if RR is a part of that solution. It seems to me to be a pretty natural fit, but suddenly it's starting to look more like Ceph or Portworx.