Jupyter images cannot be run in a secured multi tenant Docker environment.

Although you have tried to set up the Docker images to be able to run as a specific non root user, this is not sufficient in a Docker environment which provides support for multi tenant hosting.

In such an environment, as well as enforcing that images cannot be run as root, they will also run the Docker containers as different user IDs for different real users or customers, or even different applications.

Different user IDs are used to reduce the risk that were someone to break out of the Docker container, even though they wouldn't be root, that they then still cannot access the data of, or otherwise interact with, another application.

Even with what you have tried to do with ensuring that a non root user is used, it is not done in a way that a hosting environment can actually verify the container isn't running as root. This is because you use a named user for the USER statement in Docker. This says nothing about the real uid behind it, which is going to be dictated by the local passwd file within the Docker container, something which can't readily be verified by a hosting environment. A secured Docker environment is not therefore going to allow your image to run.

To work in such an environment a number of changes are required to the Jupyter images. I am going to describe the required changes here and explain why they are needed.

I am happy to provide a pull request containing all the changes, but would only do that if there is an indication that you would be willing to make the changes. If for some reason you are going to baulk on making the changes, then some other solution would need to be found. If no solution can be found that you are happy with, then the Jupyter Notebooks will not be able to be run in such environments and so you would miss out on potential users. Given that such environments are going to become more prevalent in enterprise cloud deployments where security is very important, it is a large space you would be missing out on.

Integer User ID vs Named User

The first problem as mentioned above is the use of:

USER jovyan

when closing out the Dockerfile. This results in the ContainerConfig.User setting within the Docker manifest data being jovyan.

In a hosting environment which blocks the running of Docker images as root, this image would still not be allowed to run. This is because a hosting environment cannot assume that jovyan hasn't actually been mapped to use the uid of 0 in the passwd file within the container.

To remedy this issue, the Dockerfile for each image should instead be closed out with:

USER 1000

where 1000 is the actual uid for the jovyan user. In other words, the integer uid of the UNIX user account should be used in USER and not the name of the user. That way the hosting environment can actually trust that your image is actually going to run as uid 1000.

Group That User Belongs To

The next issue is the group that the jovyan user is added to, as created by:

# Create jovyan user with UID=1000 and in the 'users' group
RUN useradd -m -s /bin/bash -N -u $NB_UID $NB_USER && \
    mkdir -p /opt/conda && \
    chown jovyan /opt/conda

This is relying on the default rules in place for the system of adding any user created to the users group.

This is problematic for the case where a hosting environment overrides what uid the container is run as. This is best explained by showing what happens.

$ docker run --rm jupyter/minimal-notebook id
uid=1000(jovyan) gid=100(users) groups=100(users)

As you can see, when no uid is specified, that specified by USER in the Dockerfile is used. Further, the user inherits the group it is associated with in the groups file.

If however the uid is overridden using the -u option to docker run we get:

$ docker run --rm -u 2000 jupyter/minimal-notebook id
uid=2000 gid=0(root) groups=0(root)

This is occurring because the uid being used doesn't correspond to an actual user account defined within the passwd file of the container. It therefore falls back to using the root group with gid of 0.

Hosting environments which use such uid overrides when running a container will actually use very high numbers for the uid, which are going to be well beyond what a passwd file might define.

The potential for a solution exists for this in the USER statement of the Dockerfile. That is that the USER statement can be given both a uid and a group. Thus you might think you could use:

USER 1000:100

Unfortunately this isn't the case as even with this the result will be:

$ docker run --rm -u 2000 my-minimal-notebook id
uid=2000 gid=0(root) groups=0(root)

$ docker inspect my-minimal-notebook | grep User
            "User": "1001:100",
            "User": "1001:100",

That is, whenever using the -u option, if only a uid is provided, docker run ignores that a group was specified by the USER statement and doesn't implicitly pass it.

This might be worked around by the hosting environment inspecting the Docker image to query the group and so calling docker run as:

$ docker run --rm -u 2000:100 my-minimal-notebook id
uid=2000 gid=100(users) groups=100(users)

but such hosting environments that I know of do not do this and so it cannot be relied upon.

Because one cannot depend on USER listing the group there is only one real solution.

This is not to add jovyan to group users but to group root. Thus when adding the user account you would use:

# Create jovyan user with UID=1000 and in the 'users' group
RUN useradd -m -s /bin/bash -N -u $NB_UID -g 0 $NB_USER && \
    mkdir -p /opt/conda && \
    chown jovyan /opt/conda

The only concern with this is that you currently advertise in the documentation for each image:

Unprivileged user jovyan (uid=1000, configurable, see options) in group users (gid=100) with ownership over /home/jovyan and /opt/conda

Although using the group root is the preferred solution, the issue would be whether some user out there has depended on the fact that you say the user is in the group users. I can't think of any reason why someone would be hardcoding the group users into anything, but you may have other requirements for it being users that I don't know of.

The only other thing one can do is leave jovyan in the users group and address one of the reasons for doing this using changes to permissions on directories and files, which is required anyway, and which is explained below. That does mean though that a different gid is used when run as jovyan vs a uid with no account. I don't know of a reason this would cause an issue in itself, but would still be preferable that it be made consistent.

Ownership of Directories and Files

Whether or not you add jovyan to the root group instead of the users group, changes would be needed to the group ownership of directories and files. What would need to be done here is to make the root group the group owner of everything under:

/home/$NB_USER

and:

/opt/conda

Further, the root group would need to be given access to all directories and ability to write to them. Similarly, for files, the group would need the ability to read and write files.

RUN chgrp -R root /home/$NB_USER \
    && find /home/$NB_USER -type d -exec chmod g+rwx,o+rx {} \; \
    && find /home/$NB_USER -type f -exec chmod g+rw {} \; \
    && chgrp -R root /opt/conda \
    && find /opt/conda -type d -exec chmod g+rwx,o+rx {} \; \
    && find /opt/conda -type f -exec chmod g+rw {} \;

This will result in the user owning everything being jovyan and the group being root.

Consider now the two scenarios above.

If the jovyan user was being added to the root group instead of users, it would have access by virtue of it being both the owner by user, as well as group membership.

If the jovyan user was left as being in the group users, it would rely on access by virtue of it being owner.

For where a uid override was now done for a uid with no user account, the gid would be that of the root group. The user in this case would have access by virtue of group membership.

Overriding the HOME Directory

A final thing that needs to be done is to force set the value of the HOME user environment variable to /home/$NB_USER. This is necessary to deal with the case where an unnamed uid is used when running the Docker container.

ENV HOME /home/$NB_USER

If this is not done HOME will default to /, but then no write access to that for the .jupyter directory would be possible.

Using a Docker Shim Layer

To demonstrate the viability of this being able to work one can create a derived Docker image which goes back and fixes up at least the directory and file permissions, plus ownership, the USER statement and the HOME environment variable.

FROM jupyter/datascience-notebook

USER root

RUN chgrp -R root /home/$NB_USER \
    && find /home/$NB_USER -type d -exec chmod g+rwx,o+rx {} \; \
    && find /home/$NB_USER -type f -exec chmod g+rw {} \; \
    && chgrp -R root /opt/conda \
    && find /opt/conda -type d -exec chmod g+rwx,o+rx {} \; \
    && find /opt/conda -type f -exec chmod g+rw {} \;

ENV HOME /home/$NB_USER

USER 1000

You can test it works by running with the default user plus an explicit uid passed via the -u option to docker run were the uid doesn't map to a user account. You need to ensure you have write access to write notebooks, plus the ability to install new Python packages through the notebook. The latter needing access to site-packages for where packages are installed, plus the pip cache.

You will with this shim still have the situation of the running gid being different between jovyan and an unnamed user, but I haven't see any issues with that at this point. Having jovyan be in group root would still be preferred.

Also note that you wouldn't want to run the changes in group ownership and permissions in a separate step to avoid addition of extra layers. They should be done to the extent necessary as part of every command when Python and/or the packages are installed. Unfortunately Docker doesn't provide a way of specifying a umask, plus some software installers don't honour umask anyway so permissions need to at least be fixed up anyway.

Reasons for Seeking Changes

As to why am pursuing it, it is because I work for Red Hat and we have our new version of OpenShift which uses Docker and Kubernetes. Because it targets the enterprise space and also multi tenant online hosting, security is very important. The default security mechanisms in Docker are not sufficient to ensure a completely secure environment in multi tenant environments, thus why the extra security provisions around assigned uid's.

I would really love to be able to demonstrate Jupyter Notebooks working using the official Jupyter images rather than having to create my own, or use a small shim Docker image to fix them up so they work. Such solutions are not practical and can't be used by real customers as they would be unsupported in any way.

In addition to at least getting the standard Jupyter images be usable in OpenShift, I would also like to provide additional changes for inclusion which Source to Image (S2I) enable the images. What S2I does is provide a way of being able to use an image as a builder, which combines files from a users Git repository with the base image to create a final deployable image. In the case of Jupyter images, this would provide a one step solution for users to combine notebooks they have with a Jupyter image for deployment without them needing to know how to write a Dockerfile themselves.

The S2I system is a standalone project and can be used by users themselves to create new Docker images, but it also has builtin support within OpenShift. This means it becomes a trivial matter for a user to deploy the Jupyter images and have their notebooks automatically copied into the image when run.

The changes required to S2I enable the images are trivial. Being the addition of 5 LABEL definitions to the Dockerfile and the addition of two shell scripts to the minimal-notebook base image. The first of these being four lines and the second two lines.

If you want to go a step further, the S2I scripts could also be extended to allow a file to be provided with a list of additional Python packages to be installed automatically as part of the deploy. In other words, similar to how a requirements.txt file for pip is automatically detected by hosting environments, with your extra packages being installed.

As far as timeline for working to come up with a solution that works for both of us, I am hoping to have something by the time of PyCon US so I can demonstrate it and talk to people interested in Jupyter deployment in the cloud to see how else we can help as far as providing an environment to run it, including helping developing solutions for provision many Jupyter instances for a classroom like teaching environment.

So let me know if you would like to try and find a solution to these issues and I can work on the pull request I mentioned if you don't wish to work it out yourself.

After reading your notes through once, I have a few questions off the top of my head and might have a few more as I re-read what you've written. Mostly, I'd like to understand if and how what you propose will affect the experience of users pulling these images and running them in a single-user environment (probably the most popular use of these images today).

How do the proposed changes for initializing the UID / GID plus permissions impact setting up permissions on host volume mounts, if at all? (I don't believe it precludes it and the steps might even bet the same as they are today.)
When you talk about setting user permissions on /opt/conda and the user home directory, you say "They should be done to the extent necessary as part of every command when Python and/or the packages are installed". Does this suggest we have to run the chmod after any conda, pip, or addition in every Dockerfile? Would sticky bits for group not work?
When you talk about hosting providers setting the UID but not the GID after inspection, does it really matter what group the user ends up running in considering its gid=0 anyway by default and the user (not group) already has ownership over all the files in the home directory and /opt/conda when -e NB_UID=2000 is passed on docker startup for the start-notebook script to reset permissions on those directories? (I'm assuming the problem here is that common cloud providers are not going to be easily configured to pass that env var.)
Is there any security implication for putting the container user in gid=0?

I am happy to provide a pull request containing all the changes, but would only do that if there is an indication that you would be willing to make the changes. If for some reason you are going to baulk on making the changes, then some other solution would need to be found. If no solution can be found that you are happy with, then the Jupyter Notebooks will not be able to be run in such environments and so you would miss out on potential users.

If we can strike a balance between how easily fly-by users can run a container on a single host, and how easily an enterprise can run containers in a secure multitenant environment, then by all means we should do that here. If not, then refining the scope and purpose of this repo becomes a question to the broader Jupyter team, and that scope should help us decide how to proceed (e.g. maybe this repo becomes the destination for end-users to get images to run as one-offs and another repo becomes the spot for maintaining images built for cloud providers).

/cc @fperez @rgbkrk @jakirkham @jtyberg @minrk

EDIT: typo

Just to add a bit to what @parente has already said with an eye to how this might be done.

Mostly, I'd like to understand if and how what you propose will affect the experience of users pulling these images and running them in a single-user environment (probably the most popular use of these images today).

:+1: Exactly. One of the huge benefits of these stacks is that it let's a user with little to no knowledge about packaging in the Python ecosystem get up and running quickly. I would really want to make sure this use case is unaffected. Think college students working on an assignment that need a notebook with the SciPy stack quick.

Related. I would also want to be very sure that conda and pip still behave nicely for the end user. Many people like to use these as building blocks where they need to have a working Jupyter Notebook with the SciPy Stack or similar.

Also, related. If we do need to make these sorts of changes to our conda and pip install steps, I would also want to make sure that we simplify this process for intermediate users who are familiar with some aspects of Docker and Python package management, but not be aware of these security concerns. In short, we should be sure we are handling cases under the hood so that Dockerfiles that install more packages after the fact automatically follow the same system. This will likely simplify our own maintenance effort with regards to these concerns as well.

Finally, any code like this must come with clear documentation. What has been written in this issue description is a good start.

Is there any security implication for putting the container user in gid=0?

This is a little worrisome to me as well.

Thanks for the detailed post!

I think @parente and @jakirkham have covered most of what's occurred to me, re: uid/name, etc.

I think the proposed groups/filesystem permissions changes should be fine, as long as the user can install packages, which is our main requirement, there.

I think the single-user case is the first priority for these images, and tmpnb/jupyterhub-type cases second. It is not our intention for these to be the way to use Jupyter in docker. These are a way, and our way, but they shouldn't be expanded to cover all possible cases. I'm not sure, yet, whether the different pieces of this proposal fall in scope or not. I'm pretty sure that maintaining enterprise-ready-multi-tenant-security is not something that we can commit to, though, unless you are proposing to take on that responsibility yourself.

The desire is that it shouldn't affect at all the typical 'docker run jupyter/minimal-notebook' experience.

In response to @parente points:

(1) For a file system mount from the Docker host, then user/group that it is running as in the container would apply. This is actually a good argument for not changing the default group from users. That way for normal Docker usage everything stays the same. The scenario of a different uid being used is only going to occur in a hosting service where they are deliberately wanting it to run as that specific uid, in which case they are getting the desired result which is that writes to mounted volumes do use the specific uid. For this special case it seems to be accepted by the hosting services that gid is going to be 0 given that there is no user account from which to inherit a different group.

(2) From my experience I don't trust sticky bits on group in Docker, ie., g+s. I know that for some reason if you try and change the pip cache directory to g+s from the Dockerfile then it breaks everything, leaving the directories with some really strange permissions.

$ docker run --rm -it my-minimal-notebook bash
I have no name!@2bbb2c46e3d6:~/work$ ls -las
total 8
4 drwxrwsr-x  2 jovyan root 4096 Mar 18 11:53 .
4 drwxrwsr-x 14 jovyan root 4096 Apr 22 12:39 ..

I have no name!@2bbb2c46e3d6:~/work$ ls -las ..
total 44
4 drwxrwsr-x 14 jovyan root 4096 Apr 22 12:39 .
4 drwxr-xr-x  9 root   root 4096 Apr 22 12:39 ..
4 -rw-rw-r--  1 jovyan root  220 Nov 12  2014 .bash_logout
4 -rw-rw-r--  1 jovyan root 3515 Nov 12  2014 .bashrc
4 drwxrwSr-x  4 jovyan root 4096 Apr 22 12:39 .cache
4 drwxrwsr-x  2 jovyan root 4096 Mar 18 11:53 .continuum
4 -rw-rw-r--  1 jovyan root   42 Mar 18 11:53 .curlrc
4 drwxrwsr-x  2 jovyan root 4096 Apr 22 12:39 .jupyter
4 drwxrwsr-x  2 jovyan root 4096 Mar 18 11:53 .local
4 -rw-rw-r--  1 jovyan root  675 Nov 12  2014 .profile
4 drwxrwsr-x  2 jovyan root 4096 Mar 18 11:53 work

I have no name!@2bbb2c46e3d6:~/work$ ls -las ~/.cache/
ls: cannot access /home/jovyan/.cache/..: Permission denied
ls: cannot access /home/jovyan/.cache/.: Permission denied
ls: cannot access /home/jovyan/.cache/pip: Permission denied
total 0
? d????????? ? ? ? ?            ? .
? d????????? ? ? ? ?            ? ..
? d????????? ? ? ? ?            ? pip

This makes it impossible to then install any further packages.

Wherever it is done, the outcome needs to be that group is root and that the root group has read/write permissions. Any fix ups would ideally only be done once, but you layer images on top of others and install additional packages, so hard to avoid going in and revalidating and changing as necessary things in a derived image. Where any fix ups are necessary or best done would come out when start looking at what would be required to change the existing Dockerfile rather than using a shim.

(3) A hosting service wouldn't be using the -e NB_UID method. They wouldn't allow the container as root such that it could work. A hosting service isn't also going to allow sudo privileges either. It therefore wouldn't be necessary to consider how that works in case where -u is used to override what user runs as. If we leave the default group as users still, that would all work the same and not be affected. All that is required to allow gid of 0 is changing group ownership/permissions of directories and files.

That gid is not inherited from USER by docker run when using -u with just a uid is a strange one. One could argue that this could be seen as a bug in docker run. But then likely because they wouldn't want to make a change to it, they will say is by design because -u is meant to replace USER, thus overriding it entirely. And since just a uid in USER results in group associated with user being used, and there is no user, then that is why ends up as gid of 0.

I have started a discussion in OpenShift about whether OpenShift should itself inherit any group from USER and add it to the -u when the uid is overridden. This would actually be the best solution as then your Dockerfile could say 1000:100 and so even though uid didn't have matching user account would run as group users and group ownership wouldn't need to change, although permissions would still to ensure that group has write access etc. I am not sure what the outcome of that discussion may be and whether they may not want to change anything.

(4) I don't know of any security implications of running as gid of 0 and security folks would have looked at that quite closely in OpenShift and given it an okay. Personally the only issue I have seen, is that some Debian packages left some documentation files they installed with group write access for root group. So you could technically modify those documentation files which were part of an installed package. Have seen no instances of any important system files having group write access to root which would cause an issue.

In response to other follow ups, the only thing I see being raised that would require some care as to how to handle it, is the case where people use the images as a base image to then install further packages. Or even where the install packages from the running notebook and then checkpoint the image. As they would be doing it as jovyan and it still has ownership of directories, it should still be able to install packages. The resulting image should also work fine for the normal run case where runs as jovyan or -e NB_UID is used, the latter modify the user somehow anyway as far as I can tell.

The issue in this case of derived images is what if they tried to take that derived image and use it with a hosting service that used -u to override the user and a further install of packages was attempted by the specified uid running as gid of 0. In this case the packages first installed in that intermediate Docker layer could end up with the wrong group since no fix up is done. Use of g+s on directories would possibly solve that, but I highlight the problems with the pip cache directory with that. Perhaps why the pip directory gets mucked up in that way needs to be understood and how to get around it so g+s can be used. Then all would be good.

I will need to think through some more on that specific case and what issues may arise. I will investigate some more what the issue is with g+s on the pip cache directory. I have seen the same issue once before with a directory under HOME which was only o+rwx and used to store some nssdb file. It is something to do with Docker layers. In the prior case I saw it, the problems with bad permissions went away when Docker squash was used to remove layers, but running that is not a solution. They did solve it in some other way, so need to ask what they did.

So let me go sort out the sticky bit issues for the pip cache directory and whether can work out how g+s can be used and whether having that helps in any way.

So they actually solved the prior nssdb problem and strange permissions by recreating the directory which caused the problems.

https://github.com/openshift/s2i-python/issues/60#issuecomment-203173704

This didn't involve g+s but was triggered in a different way. I will try recreating the pip cache directory with permissions which have it as group writable to start with. If pip decides it doesn't like that and reverts the permissions that will be a problem. That will be harder for me to test though as have to add pre-create in base image Dockerfile.

Not sure whether this is part of the problem or not, but have you tried different union filesystems with regards to this sticky bit issue?

Thanks @GrahamDumpleton for your attention to this topic. I am not a security expert, but I have spent (wasted) a fair amount of time struggling with setting up non-root users within Docker containers, and still have the containers be usable. Has anyone on your team experimented with Docker user namespaces?

I ask because dealing with all this non-root user business within the container seems like a ton of accidental complexity. It would be so much easier if one could just be root within the container, I think both from a user's perspective and from a maintainer's perspective (on this repo, in particular). Sometimes I wonder what would happen if one day we decided to change the notebook user from jovyan to something else.

This has been a good discussion back and forth. For a way ahead, there are at least components out of here that can be broken up into smaller PRs. One of these, that I was a bit surprised we didn't already have, is setting ENV HOME. That's done in the demo images (separate repo).

@jtyberg My understanding is that right now Docker user namespaces is not sufficient, or cannot be applied in a practical way, for a multi tenant system. Part of the issue is:

""" When user namespace support is enabled, Docker creates a single daemon-wide mapping for all containers running on the same engine instance. """

So although it can remap uid ranges such that you could allow root inside of the container, the same uid range would need to be used by all containers running on the host.

This means that containers for different users cannot use a different range of uid's at the host level.

Thus if someone can break out of the container, even though not root at that point, they could potentially see file system data from a different customer, or perhaps interact with customer applications through file system sockets where user based access control is used.

To solve this problem properly such that different containers could be mapped to different uid ranges for file system storage at least, I am told that it would require kernel changes and even if that was something that kernel developers would be prepared to do, that would likely be a long time coming.

One promising thing to mention is that I actually got a favourable response on OpenShift being changed to honour the group specified in USER statement of Dockerfile when overriding the uid. I need to read the responses I got properly, but if that can be done, that I believe would provide the cleanest solution and only need to ensure that files/directories have appropriate group level rwx as necessary. That is, group could still be users. So I will go down that path a bit and see what I can work out.

The other technical point of relevance with user namespaces is simply that they're still a relatively new security mechanism at the kernel level, and much harder to systematically audit for potential escape hatches (i.e. using root access in one container to gain root access either on the host or inside a different container), as compared to enforcing a blanket ban on the use of the root UID inside tenant containers.

However, if OpenShift can be updated to honour the group ID in USER statements, that does sound promising. Am I right in thinking that the remaining change requests in that case would be to:

Update the final USER statement in the base Dockerfile to be USER 1000:100 rather than USER jovyan
Set the HOME env var appropriately

If so, then it would make sense to submit at least those changes, while the discussion about using the default root group vs the Dockerfile specified group can continue on the OpenShift side.

Plus:

Ensure directories/files are group readable/writeable/searchable as appropriate.

Having OpenShift honour the group may take a while to achieve as looks like that will entail a change to Kubernetes.

You don't per chance know whether it is possible to modify the system wide behaviour that root group is used for gid when process started with uid not in passwd file? If could change it to use users group instead that would also solve the problem. I can't see anything in /etc/login.defs.

FYI. One can rule out using ACLs. AUFS doesn't support them. So cannot do:

RUN setfacl -Rdm g:root:rwx /home/$NB_USER

In respect of pip cache permission issues, why is the cache not being disabled in the first place?

When building Docker images, one would normally use --no-cache-dir or set environment variable PIP_NO_CACHE_DIR as there is no need to waste space keeping the cache directory around as you are not going to be needing to reinstall them a subsequent time into a fresh virtual environment. Instead they would be installed once for the life of that image.

And finally found the comment I thought I had seen before which acknowledges the permissions issue on the pip cache directory is an AUFS issue, one which apparently still hasn't got a resolution.

https://github.com/docker/docker/issues/783#issuecomment-19237045

This means any permissions must always be fixed up in same layer, or for the pip cache, pre create it with more open permissions before any pip install is run.

Or as I note above, disable the pip cache anyway as there is no good reason to have it that I have ever seen for a Docker image.

Docker images can also be used to bootstrap mutable local environments, and in those cases you want more desktop-like behaviour (such as a writable pip cache).

If you explicitly use --no-cache-dir on installs in base images and not use an environment variable in the image, then you can still use the pip cache directory in such mutable environments. This is because the cache directory wouldn't exist at that point and so would be created so long as home directory is group writable. The directories have to be group writable anyway.

One of these, that I was a bit surprised we didn't already have, is setting ENV HOME. That's done in the demo images (separate repo).

@rgbkrk We are currently relying on HOME to be set by virtue of running as jovyan in the image. In any of the stacks, if you do:

!env

in a Python notebook, you'll see HOME=/home/jovyan.

I think setting HOME in the Docker environment means that when you're running any commands as root, you're still pointing to the jovyan home directory. Not sure that's desirable.

It becomes important if you want to pre populate an image with configuration files. You don't want to have put in place /home/jovyan/.jupyter and have it ignored when you run as root. The simplest way to avoid that is to override HOME.

@GrahamDumpleton Thank you for addressing my questions.

I don't know of any security implications of running as gid of 0 and security folks would have looked at that quite closely in OpenShift and given it an okay. Personally the only issue I have seen, is that some Debian packages left some documentation files they installed with group write access for root group

I was wondering about the implications of membership to root group if a user breaks out of the container.

So let me go sort out the sticky bit issues for the pip cache directory and whether can work out how g+s can be used and whether having that helps in any way.

I think we can disable the pip cache during the Docker build like you say. I'm more interested in if sticky bits can be used on /opt/conda successfully or not so that owner/permission touchups don't have to be applied after every pip/conda command.

I foresee difficulties explaining the find+chmod requirement to every new contributor to the stacks. I also see us forgetting about the requirement during code review and causing regressions unless there are some automated tests put in place.

I think @rgbkrk is right that the next step is to start opening small PRs that address some of the easiest issues first (e.g., disable pip cache during the docker build, set ENV HOME). We can build up to the final solution in increments, and sanity check the impact of each change to the single-user experience along the way.

+1 to breaking this up into small PRs that start with the easier pieces, and thanks a lot to @GrahamDumpleton for the thorough work above!

Pinging @freeman-lab, who might be interested in this from the perspective of current and future work on Binder...

1-year check-in. A few PRs last year addressed low-hanging fruit. I see https://github.com/GrahamDumpleton/openshift3-jupyter-stacks exists with Dockerfiles that touch up permissions starting FROM the images here. Looks like a clean solution to me.

@GrahamDumpleton I'm going to close this issue as inactive. If you think what you have in your openshift3-jupyter-stacks repo can merge here without impacting non-OpenShift use, feel free to submit PRs.

I have not created any PRs against your repos because there needed to be a decision made up front on one key issue. I wasn't going to do a lot of work creating a PR against all the different images before the decision was made on the strategy as would be quite a bit of wasted effort on making changes and testing it given wouldn't likely be accepted at this point.

The specific issue which needs to be looked at further is that of changing the group for files, directories and the running process to be the group root (gid of 0). All files and directories also need to be group writable where appropriate.

So far in the discussion this was a sticking point.

Fixing up these permissions in a derived Docker image as was being done in openshift3-jupyter-stacks is not a workable solution because of how Docker images work. This is because the resulting images became even fatter as 400MB was added in a single Python version image, and 800MB for dual Python version image. This is because changing permissions on files in a new layer causes a new copy of the file to be made in a new layer. The resulting images were getting so fat as to become unusable in scenarios.

I will review where suggested changes were at as far as what has already been done and create a list of what is outstanding. At this point though my general recommendation have been giving people is to ignore the Jupyter Project images.

jupyter / docker-stacks