Automatically resolve permissions for Linux

[jasonmccallister]

Description

When running Nitro (or any container with a bind mount) the user often has to manually set permissions based on the host user and container user.

With 3.0 we have changed the image for Alpine based to Debian and this "breaking" change is a good time to work on resolving the automatic file permissions issue on Linux based systems.

There are a few approaches:

1. Adding a permissions command to make this easier #358
2. Dynamically grab the host uid and guid and run a command in the container as root (not the best approach)
3. Consider using NFS on linux based machines which allows you to specify the uid/guid on creation https://docs.docker.com/storage/volumes/#create-a-service-which-creates-an-nfs-volume

This is a known issue with Linux and containers, but I think there is a programatic way to resolve this for Nitro users.

[xavierdemoor]

I'm in to help testing 3.0 under Windows WSL and give all the help I can :)

I will write another possibility later today when I will have time for it.

[josephkerkhof]

This is one of the main features preventing us from using Nitro as our dev environment. I'd be happy to test on my Linux machine when this is ready.

[jasonmccallister]

I did a little research (during an internet outage) and found a couple of things that may be helpful. Using this article to look at some of the ways other linux users have resolved this issue, the last option talks about modifying the Docker daemon configuration to tell Docker to enable user namespace remapping with the userns-remap configuration in the Docker daemon full configuration reference.

The TL;DR is to perform the following tasks:

• Create or modify the daemon.json file to include the userns-remap to the username on the host machine (e.g. jason).
• Restart Docker, maybe manually remove a site container and run nitro apply - this is just to ensure the mount is configured with the new setting (not sure if this is needed or not)
• Try accessing the site and modifying the files - without changing ownership or permissions

Is anyone willing to give this a try on the current version of Nitro? I will be trying this when I can get to my desktop linux computer but might be a while.

[josephkerkhof]

I tried what you suggested @jasonmccallister, but I'm receiving a different permission error now:

Error: unable to start the container, Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "/home/me/path/to/craft" to rootfs at "/app" caused: stat /home/me/path/to/craft: permission denied: unknown

I did try removing all remaining containers and running nitro apply as suggested, but the error remains.

[xavierdemoor]

@jasonmccallister Is this modifying the Docker general behavior globally? If yes, it's not a solution. It can cause border effects for other containers not related to Nitro. I don't think it's a wise idea to alter Docker behaviors.

I still think the best approach is to run container services with the user ID in use when installing Nitro.

Nitro is a local dev environment. I can understand it's painful to maintain multiple Dockerfile, but it's the simplest solution to remove all these permissions problems.

After thinking about the "let user bring their own image", it's not that difficult to do too. Nitro can run as now and user can override the image to use. It's their problem to build a correct image to fit with some Nitro's restrictions like exposed ports if they want to use their own images. The community can also provide and maintain some images alternatives in an official Craft repo.

At the end, what I like in Nitro is to provide me easy way to have https, adding site, db engine, redis,... The "web" image (nginx php, apache php,...) needs to mirror the hosting provider as much as possible, so it can be the responsibility of the developper, or just using the default craft image if they don't want to.

[jasonmccallister]

If yes, it's not a solution. It can cause border effects for other containers not related to Nitro. I don't think it's a wise idea to alter Docker behaviors.

I partially agree with you. However, the article talks about running as root and this is not an issue and talks about an approach to fix it that is not Nitro specific. I don't like recommending changing the daemon to run, but its a setting that exists for a reason (just not sure about the actual use case).

run container services with the user ID in use when installing Nitro.

The real issue is determining what user to run these as. Which that article points out also needs to exist in the container. That does not mean we can't tell the container to run as that user and maybe figure out the container user problem programmatically. If we ran as root, this would not been an issue, but that is a bad practice. Even though this is a local development environment, I would like the images to be close to "production" so there is that same experience and environment from development to production.

After thinking about the "let user bring their own image", it's not that difficult to do too.

While it's not that difficult to do, that's not an approach everyone needs. Maybe we do take this approach, but I think resolving the permissions issue without telling Linux users "oh by the way you have to do this" provides a good first use experience. Nitro tries to take the complexities of Docker and make it simple to use without learning all of the jargon.. We should eventually support the "bring your own Dockerfile" feature, but I don't think that is the proper solution for this specific issue. Solving this problem and then adding that feature would be a better approach, that way Linux users get a good out-of-the-box experience to getting started with local development.

At the end, what I like in Nitro is to provide me easy way to have https, adding site, db engine, redis,... The "web" image (nginx php, apache php,...) needs to mirror the hosting provider as much as possible, so it can be the responsibility of the developper, or just using the default craft image if they don't want to.

This is ultimately the goal we are working towards and appreciate everyone's feedback! Don't take our questions/comments as "anti-feature requests", we just want to ensure we solve the right problems and everyone benefits from the changes (which is especially tricky when working across multiple operating systems).

We will get there, but might have to take a few detours and trial and errors to get the best solution with minimal changes/tweaks.

[jasonmccallister]

@musicaljoeker we are going to try to set this up internally and see if we can get it working..

[xavierdemoor]

Is anyone willing to give this a try on the current version of Nitro? I will be trying this when I can get to my desktop linux computer but might be a while.

I tried it on WSL, it just crash Docker (and doesn't allow me to revert the change, so Docker is still crashing). See this issue https://github.com/docker/for-win/issues/6897

[xavierdemoor]

The real issue is determining what user to run these as. Which that article points out also needs to exist in the container. That does not mean we can't tell the container to run as that user and maybe figure out the container user problem programmatically. If we ran as root, this would not been an issue, but that is a bad practice. Even though this is a local development environment, I would like the images to be close to "production" so there is that same experience and environment from development to production.

You don't use the exact same image in dev than in production. Like on docker craft repo, you have a prod image and a dev image. You can mimic as much as possible prod environment, I don't see where is the problem to add the trick in the dev's image.

This trick is meant to solve local dev issues with file permissions only. My actual dev and prod images are exactly the same with two exceptions. I included dev tools into my dev's image AND I added this uid/gid trick to avoid files permissions. But they are exactly the same at these exceptions. Same Nginx version, same PHP version, same PHP extensions, same services configurations.

IF (like said in the article) you're developping on two machines, you have two computers, Nitro will be set to the correct uid on both machines because it will build the dev image according to the uid. It's the one of the reason the author gave to use another method than running services under the host uid.

Author:

The problem with this approach is that is not portable. What if I am developing using more than one computers where in each computer my user has different ID ? Or what if many developers use this image for development ?

For the second, the devs will use the same Dockerfile, just not the same images. Each dev image is built during Nitro's install or Nitro's update. The only difference will be the uid/gid passed during the image's build. If they don't build their own images, I don't understand the problem too. Each image on each dev's desktop will be different only for the uid running the services.

Don't take my words as "I'm right, you need to do that". If you find a better way to deal with this problem, without altering Docker's behaviors, I will be very happy. I just think there is a reason why Linux/WSL people are using this trick than another. ^^

About altering Docker's behaviors/config, I had an overview of what happen when trying a not-supported feature on Docker for WSL :P I needed to reset to factory my Docker to be able to run it again. It removed all my images, containers, volumes, ... There is probably a way to do it without resetting everything, but at this point it was not a real deal as my desktop is fresh installed. On WSL, we don't have a file to edit to change the Docker settings. It's somewhere on Windows, I think.

[jasonmccallister]

I just merged https://github.com/craftcms/nitro/pull/392 which adds a new base image and specifically includes a nitro user with the uid 1000. We (@angrybrad, @timkelty and I) tested the 3.0 branch with the new image on a WSL2 machine and it resolved the issue. The reason this worked is because the user ID on the host machine was 1000 and the containers user ID was 1000.

This will probably cover 80 percent of the use cases as most distributions start at user 1000 but that is not always the case and we still need to solve for those users who have different user IDs but we are a step closer without modifying the Docker Engine (which I agree with @xavierdemoor is not a good approach).

We're making progress, slowly but surely!

[xavierdemoor]

@jasonmccallister

This will probably cover 80 percent of the use cases as most distributions start at user 1000 but that is not always the case and we still need to solve for those users who have different user IDs but we are a step closer without modifying the Docker Engine (which I agree with @xavierdemoor is not a good approach).

I didn't have time to check all PR in your merge, but I saw the image have a $IMAGE_USER and use fixed uid and gid. Can I propose to use a fixed username but dynamic uid/gid ? The username doesn't really matter; it can be named x, y or z, only the uid/gid matter.

ARG UID=1000
ARG GID=1000
...
RUN mkdir /app && \
    addgroup www --gid ${GID} && \
    adduser www --home /app --uid ${UID} --gid ${GID}--no-create-home && \
    chown -R www:www /app
...
USER www

So we'll be able to do something like: nitro rebuild 1000:1000 or nitro rebuild and internally do a docker build --build-arg UID=$(id -u) --build-arg GID=$(id -g) ? if 1000:1000 isn't the good user id. It will rebuild the image locally for this particular case.

[clarknelson]

I assume that we may be able to get a fix to this issue in nitro 3.0?

[clarknelson]

any update on this issue?

[clarknelson]

I guess my biggest issue with using Craft and Nitro on Ubuntu is writing to storage, cpresources, and (especially) project config. Other than that it's a dream, I can get set up with a repo in 15 minutes. Unfortunately as the fix currently stands (afaik), 777 these directories in the repo doesn't seem wise. Is there anything I can test? I am starting a new project and am almost to the point of buying a mac because I want to use Nitro for development but these issues are pressing on my mind.

Kindest regards, Clark

[proimage]

Hey Clark, I've gotten Nitro to the point where it works well for me under WIndows -> WSL2 -> Ubuntu. For permissions, I made a little script called nitro-perms.sh that I put in the root of my projects (and have added to .gitignore, of course).

#!/bin/bash
chmod 777 ./.env
chmod 777 ./composer.lock
chmod 777 ./composer.json
sudo chmod -R 777 ./config
sudo chmod -R 777 ./storage
sudo chmod -R 777 ./vendor
sudo chmod -R 777 ./web

I suspect that 777 in Nitro might not matter, since it's only for local development...?

Anyway, HTH!

[xavierdemoor]

Hey Clark, I've gotten Nitro to the point where it works well for me under WIndows -> WSL2 -> Ubuntu. For permissions, I made a little script called nitro-perms.sh that I put in the root of my projects (and have added to .gitignore, of course).

#!/bin/bash
chmod 777 ./.env
chmod 777 ./composer.lock
chmod 777 ./composer.json
sudo chmod -R 777 ./config
sudo chmod -R 777 ./storage
sudo chmod -R 777 ./vendor
sudo chmod -R 777 ./web

I suspect that 777 in Nitro might not matter, since it's only for local development...?

Anyway, HTH!

It matters, even in local dev, because the permissions will be set to 777 in the git repo of your project.

Having to chmod for local and chmod back for deployment is not a good practice. If you (or someone else) doesn't set proper permissions for prod deployment, you're in deep shit.

[clarknelson]

proimage, I believe that 777 will give everyone access to read / write / execute the file. I think it's only a real issue when your server has been compromised and an unauthorized user has access, because then they will be able to read / write / execute all of the files.

I assume people could do tricky stuff to the storage folder / web folder / vendor folder. Even just read access on the .env file is sketchy, because the only person with access to that should be you and maybe your user group. I am not an expert so I hope I am not getting things wrong, and I don't understand everything you could exploit with 777, but I understand enough to know that your site is a lot easier to hack with those files as 777.

I had the same idea as you, as locally it is not as big a deal. But since git tracks file permissions, any changes to those will trigger a change in git. And since pretty much every deploy script out there uses git, it will find it's way to the server eventually.

I think we will need to tunnel some sort of authorization to the host machine to write files on the nitro machine. (obviously people have already though of and worked on this) I have tried to follow some of the advice in the thread but it doesn't work very well for me and I would like to avoid things like creating an arbitrary user on my Linux install just to run the web process.

[proimage]

It matters, even in local dev, because the permissions will be set to 777 in the git repo of your project.

Hmm. I think that, because most of my work was under XAMPP / Windows before Nitro came along, I must have defaulted my git install to always ignore file permission changes. Not doing so resulted in hundreds or thousands of "changed" files where the only change was the normal permissions adaptation that happens whenever Git ports files from a Linux filesystem to a Windows filesystem. In any case, my local Nitro files' permissions aren't getting transferred to the remote system. ¯\_(ツ)_/¯

Even just read access on the .env file is sketchy, because the only person with access to that should be you and maybe your user group.

The .env file should NEVER make it into any repos. Use .env.example if you need to distribute the names of the various environment variables the site expects to have on-hand in .env, just don't assign them to anything.

[clarknelson]

That's true, I forgot .env is typically not checked into version control. Was the git change local? I will need to investigate that further, it may help me move forward at least.

[proimage]

That's true, I forgot .env is typically not checked into version control. Was the git change local? I will need to investigate that further, it may help me move forward at least.

https://stackoverflow.com/questions/1580596/how-do-i-make-git-ignore-file-mode-chmod-changes/1580644

[clarknelson]

Thank you, that will make my life much more pleasant. I wonder why that is default true... git is confusing enough for new developers.

[xavierdemoor]

Hmm. I think that, because most of my work was under XAMPP / Windows before Nitro came along, I must have defaulted my git install to always ignore file permission changes. Not doing so resulted in hundreds or thousands of "changed" files where the only change was the normal permissions adaptation that happens whenever Git ports files from a Linux filesystem to a Windows filesystem. In any case, my local Nitro files' permissions aren't getting transferred to the remote system. ¯_(ツ)_/¯

It's still a bad practice to set 777 permissions on git's files. Imagine someone else work with you on a project and didn't do the trick with git permissions.

I always start with that in mind. I can start alone a personal project or client project, but I never assumed it will be forever. It's too easy to forget something like that. Even if I launch a script to set proper permissions in my deployments, I don't want 777 files in my repo. I saw too many critical problems with dev's bad habits.

If you don't care, it's your problem. I'm just a maniac on security risks. :p

[xavierdemoor]

That's true, I forgot .env is typically not checked into version control. Was the git change local? I will need to investigate that further, it may help me move forward at least.

https://stackoverflow.com/questions/1580596/how-do-i-make-git-ignore-file-mode-chmod-changes/1580644

As the answered said, it's a bad practice to use filesMode ;)

[clarknelson]

I'd love to avoid having to use 777, unfortunately it seems to be the supported fix as of right now. I assume all devs are crazy busy already with more critical fixes. At least I can use fileMode to work on this contract.

[jasonmccallister]

Sorry for the long delay, we have resolved this in the 3.0 branch and it involved a breaking change (changing the container images from Alpine to Ubuntu based). This solve a ton of other issues and one of those was setting a nitro user in the container image where the uid matches the default user on most linux machines (user 1000). This removes the need for setting any file permissions on Linux.

Right now we are focusing on Craft 4 and plan on switching gears to 3.0 development after 4.0. Nitro 3.0 is nearly feature complete and half of the team is using it internally so we feel pretty great about the changes. However, there are still a few items that are not done and are going to be cut from the 3.0 release as they are more nice to haves than needed.

We are really excited to get 3.0 out the door and in use and will make sure we keep you updated soon!

[clarknelson]

Thank you for the update, very excited to see more about the new versions :)

[proimage]

half of the team is using it internally so we feel pretty great about the changes

Dare I ask how many of them are on Windows?

[xavierdemoor]

half of the team is using it internally so we feel pretty great about the changes

Dare I ask how many of them are on Windows?

I'm not part of the team but I'm using Nitro 3 on Windows. On WSL2 to be precise.

[clarknelson]

I had an easier time using Nitro with Ubuntu so I switched to that.

[jasonmccallister]

Hi, we are closing this issue as we have decided to retire Nitro, so no additional work will occur on this project. You can read the official blog post here https://craftcms.com/blog/retiring-craft-nitro. We appreciate everyones feedback and involvement and we look forward to refocusing our efforts on Cloud!

If you're looking for a new local development environment, we recommend DDEV and have a knowledge base article to help you with the transition: https://craftcms.com/knowledge-base/migrating-from-craft-nitro-to-ddev.