search

docker / desktop-linux

Bug reports for Docker Desktop for Linux
https://docs.docker.com/desktop/linux/
81 stars 6 forks source link

Can't use chown inside bound volumes #9

Open nicekiwi opened 2 years ago

nicekiwi commented 2 years ago

Expected behavior

I should be able to run a command like chown -R 1000:www-data inside a container with a bound volume.

Actual behavior

/app # chown 1000:82 test.txt 
chown: test.txt: Invalid argument

Information

Steps to reproduce the behavior

docker run -it --rm -v /home/ezra/code/empty-dir:/app busybox:latest

/ #
/ # cd app
/app # touch test.txt
/app # chown 1000:82 test.txt
chown: test.txt: Invalid argument

The same applies in my docker-compose file:

version: '3'
services:
    php:
        image: nicekiwi/php:1.2.1
        restart: unless-stopped
        volumes:
            - $PWD:/srv/app:delegated
        expose:
            - 9000

I get the same result trying to chown anything inside it, and when running the PHP app it tries to write to cache with file_put_contents(/srv/app/storage/framework/sessions/8mFjoS66fKFGwikzIrQKsFstpbjKp1nT82H1zTYS): Failed to open stream: Invalid argument

p1-0tr commented 2 years ago

Thanks for reporting this issue :) We'll investigate and get back to you as soon as we have a fix for this.

p1-0tr commented 2 years ago

hi @nicekiwi, sorry for the long wait, we've got a candidate fix for this issue. Here - https://desktop-stage.docker.com/linux/main/amd64/78870/docker-desktop-4.8.0-amd64.deb - you will find a dev build with the fix in it. Let me know it resolves the issue for you.

nicekiwi commented 2 years ago

@p1-0tr that looks to have solved the issue 🤩 ty

cardonator commented 2 years ago

any timeline on which this will be packaged into a release? It's a major regression from Docker Engine to Desktop.

p1-0tr commented 2 years ago

@cardonator it will be released in 4.9, which is scheduled to come out at the beginning of June

DonRichards commented 2 years ago

@p1-0tr Is there a 4.9 release candidate available yet?

p1-0tr commented 2 years ago

@DonRichards there are no RCs yet, but, if you are willing, you can give the latest dev build a go:

smaccona commented 2 years ago

I suspect this is related to #17 as well, so tagging it here.

DonRichards commented 2 years ago

After installing it and running a alias to nuke all containers/volumes/etc 

docker kill $(docker ps -q) ; docker rm $(docker ps -a -q) ; docker rmi $(docker images -q) ; docker system prune --volumes -a -f ; docker network prune -f && docker network create gateway

Then I got a white screen from Docker Desktop 2022-05-27_16-09 This didn't prevent it from running docker commands but I suspect this isn't normal behavior. I can tell you the initial fetch and build seemed a bit faster.

cardonator commented 2 years ago

@DonRichards there are no RCs yet, but, if you are willing, you can give the latest dev build a go:

@p1-0tr I tried out this build and it still doesn't work correctly. In Linux, there are stricter permissions rules about bind mounting folders from the host. The permissions set translate through the bind mount and I get permissions errors in other containers. To circumvent this, we have a script that fixes permissions on the folders before running docker compose which bind mounts the folders. I end up running something like:

docker run --rm -it -v "$PWD/parentdatafolder:/code" alpine:latest ash -c "chown 999:999 /code/datafolder1; chown 997:997 /code/datafolder2;"

Prior to the 4.9.0 dev build, this command would just return the error listed above in this issue.

After the 4.9.0 dev build, the command works just fine, but my folders have the following permissions:

$ ls -la ./parentdatafolder
total 16
drwxrwxr-x  4 myuser myuser 4096 May 31 17:14 .
drwxrwxr-x 32 myuser myuser 4096 May 31 17:14 ..
drwxrwxr-x  2  100998  100998 4096 May 31 17:14 datafolder1
drwxrwxr-x  2  100996  100996 4096 May 31 17:14 datafolder2

When I then mount those folders in my compose file, they have the wrong owner and I still can't read/write to them. Maybe it's also worth noting that this works just fine in Docker Engine and in WSL2 on Windows.

cardonator commented 2 years ago

I did just test this on the final build of 4.9.0 and the above functionality works properly inside Docker despite setting wonky permissions in the host system. What is the purpose behind not setting the same permissions? Is that a Qemu thing?

DonRichards commented 2 years ago

@cardonator I don't think it's specific to qemu. I'm not using that. What I'm building is a Drupal server. The issue is the files need to be owned by nginx:www-data and not my local machine. So switching permissions is pretty common and is a necessity for normal operations and security within the container. I can confirm this does work as expected within docker CLI but not with Linux desktop. It even works fine on the Mac version of Docker desktop.

p1-0tr commented 2 years ago

@cardonator and @DonRichards, the weird permissions are an effect of how file sharing is implemented. Basically Docker Desktop is supposed to be able to run without elevated privileges, so you can't arbitrarily change ownership of files on the underlying system. This is why the file sharing server runs in a user namespace which maps user and group ids inside this namespace (seen by the VM running docker desktop) to a range of "subordinate" ids the given user is permitted to use (configured by default via /etc/subuid and /etc/subgid`).

@DonRichards - do I understand correctly that your use case would require the same view of permissions on the host and within the Docker Desktop VM?

kowiste commented 2 years ago

I just install the 4.9.0 but still have the same problem:

2022-06-14 10:05:19+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.8.3+maria~jammy started.
find: '/var/lib/mysql/ms_debit': Permission denied
find: '/var/lib/mysql/sys': Permission denied
find: '/var/lib/mysql/performance_schema': Permission denied
find: '/var/lib/mysql/mysql': Permission denied
find: '/var/lib/mysql/ms_card': Permission denied
find: '/var/lib/mysql/ms_payment': Permission denied
chown: changing ownership of '/var/lib/mysql/': Operation not permitted
chown: changing ownership of '/var/lib/mysql/ib_buffer_pool': Operation not permitted
chown: changing ownership of '/var/lib/mysql/aria_log.00000001': Operation not permitted
chown: changing ownership of '/var/lib/mysql/ms_debit': Operation not permitted
chown: changing ownership of '/var/lib/mysql/sys': Operation not permitted
chown: changing ownership of '/var/lib/mysql/ib_logfile0': Operation not permitted
chown: changing ownership of '/var/lib/mysql/performance_schema': Operation not permitted
chown: changing ownership of '/var/lib/mysql/mysql': Operation not permitted
chown: changing ownership of '/var/lib/mysql/ms_card': Operation not permitted
chown: changing ownership of '/var/lib/mysql/mysql_upgrade_info': Operation not permitted
chown: changing ownership of '/var/lib/mysql/ibdata1': Operation not permitted
chown: changing ownership of '/var/lib/mysql/aria_log_control': Operation not permitted
chown: changing ownership of '/var/lib/mysql/ms_payment': Operation not permitted
chown: changing ownership of '/var/lib/mysql/multi-master.info': Operation not permitted

If I run this container in the terminal before start docker desktop runs without any problem:

version: "3.8"
services:
  mariadb:
    image: mariadb
    container_name: mariadb
    environment:
      MYSQL_ROOT_PASSWORD: pass
      MYSQL_DATABASE: mybb
      MYSQL_USER: mybb
      MYSQL_PASSWORD: pass
    networks:
      - web
    restart: on-failure
    ports:
      - 3306:3306
    # volumes:
      # - ./mariadb:/var/lib/mysql
networks:
  web:
    external: true

For run it I just use type docker compose up -d in both cases.

p1-0tr commented 2 years ago

@Kowiste, sorry for the inconvenience and thanks for providing info on the error you are seeing. What I am pretty sure is going on is that files in the folder you are sharing don't belong to the user you are running as.

E.g. if you ran with the native engine first, the files would get chowned when the mariadb container inits, and as the native docker engine (most commonly) runs with an elevated set of privileges it will be able to modify ownership of files to users/groups that are valid on your host (outside of the container). On the other hand Docker Desktop runs as your local user. So if the files inside ./mariadb are owned by mysql, a chown inside a container running in Docker Desktop for such a file will be rejected.

TL;DR - could you try running with a fresh ./mariadb (or ensure all files in it are owned by your user)?

kowiste commented 2 years ago

@Kowiste, sorry for the inconvenience and thanks for providing info on the error you are seeing. What I am pretty sure is going on is that files in the folder you are sharing don't belong to the user you are running as.

E.g. if you ran with the native engine first, the files would get chowned when the mariadb container inits, and as the native docker engine (most commonly) runs with an elevated set of privileges it will be able to modify ownership of files to users/groups that are valid on your host (outside of the container). On the other hand Docker Desktop runs as your local user. So if the files inside ./mariadb are owned by mysql, a chown inside a container running in Docker Desktop for such a file will be rejected.

TL;DR - could you try running with a fresh ./mariadb (or ensure all files in it are owned by your user)?

Ok I just delete the mariadb folder and only have the docker compose. Still happen the same, on terminal works and now when run with docker desktop give this log:

2022-06-14 12:33:14+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.8.3+maria~jammy started.
2022-06-14 12:33:15+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2022-06-14 12:33:15+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.8.3+maria~jammy started.
2022-06-14 12:33:15+00:00 [Note] [Entrypoint]: Initializing database files

Installation of system tables failed!  Examine the logs in
/var/lib/mysql/ for more information.

The problem could be conflicting information in an external
my.cnf files. You can ignore these by doing:

    shell> /usr/bin/mariadb-install-db --defaults-file=~/.my.cnf

You can also try to start the mysqld daemon with:
2022-06-14 12:33:15 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit, ulimit -l, or https://mariadb.com/kb/en/systemd/#configuring-limitmemlock under systemd (262144 bytes required)
2022-06-14 12:33:15 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF
2022-06-14 12:33:15 0 [ERROR] InnoDB: The Auto-extending data file './ibdata1' is of a different size 0 pages than specified by innodb_data_file_path
2022-06-14 12:33:15 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2022-06-14 12:33:15 0 [ERROR] Plugin 'InnoDB' init function returned error.
2022-06-14 12:33:15 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2022-06-14 12:33:15 0 [ERROR] Unknown/unsupported storage engine: InnoDB
2022-06-14 12:33:15 0 [ERROR] Aborting

    shell> /usr/sbin/mariadbd --skip-grant-tables --general-log &

and use the command line tool /usr/bin/mariadb
to connect to the mysql database and look at the grant tables:

    shell> /usr/bin/mysql -u root mysql
    mysql> show tables;

Try 'mysqld --help' if you have problems with paths.  Using
--general-log gives you a log in /var/lib/mysql/ that may be helpful.

The latest information about mysql_install_db is available at
https://mariadb.com/kb/en/installing-system-tables-mysql_install_db
You can find the latest source at https://downloads.mariadb.org and
the maria-discuss email list at https://launchpad.net/~maria-discuss

Please check all of the above before submitting a bug report
at https://mariadb.org/jira
p1-0tr commented 2 years ago

@Kowiste - this is a bit more worrying. I was able to reproduce the issue and work around it by creating the mariadb folder, before running compose:

docker compose down
mkdir mariadb
docker compose up -d

So the issue is down to sharing a folder that does not exist, which should work fine. Thanks for reporting this, hope the work-around unblocks you while we work on a fix.

kowiste commented 2 years ago

@Kowiste - this is a bit more worrying. I was able to reproduce the issue and work around it by creating the mariadb folder, before running compose:

docker compose down
mkdir mariadb
docker compose up -d

So the issue is down to sharing a folder that does not exist, which should work fine. Thanks for reporting this, hope the work-around unblocks you while we work on a fix.

Ok yeah creating the folder first works thanks.

p1-0tr commented 2 years ago

Sorry for the long wait, fix for auto-creating folders when bind mounting a volume is lined up. In the meantime, if you are willing to give it a go here are dev-build links:

muzzwood commented 2 years ago

I just install the 4.9.0 but still have the same problem:

2022-06-14 10:05:19+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.8.3+maria~jammy started.
find: '/var/lib/mysql/ms_debit': Permission denied
find: '/var/lib/mysql/sys': Permission denied
find: '/var/lib/mysql/performance_schema': Permission denied
find: '/var/lib/mysql/mysql': Permission denied
find: '/var/lib/mysql/ms_card': Permission denied
find: '/var/lib/mysql/ms_payment': Permission denied
chown: changing ownership of '/var/lib/mysql/': Operation not permitted
chown: changing ownership of '/var/lib/mysql/ib_buffer_pool': Operation not permitted
chown: changing ownership of '/var/lib/mysql/aria_log.00000001': Operation not permitted
chown: changing ownership of '/var/lib/mysql/ms_debit': Operation not permitted
chown: changing ownership of '/var/lib/mysql/sys': Operation not permitted
chown: changing ownership of '/var/lib/mysql/ib_logfile0': Operation not permitted
chown: changing ownership of '/var/lib/mysql/performance_schema': Operation not permitted
chown: changing ownership of '/var/lib/mysql/mysql': Operation not permitted
chown: changing ownership of '/var/lib/mysql/ms_card': Operation not permitted
chown: changing ownership of '/var/lib/mysql/mysql_upgrade_info': Operation not permitted
chown: changing ownership of '/var/lib/mysql/ibdata1': Operation not permitted
chown: changing ownership of '/var/lib/mysql/aria_log_control': Operation not permitted
chown: changing ownership of '/var/lib/mysql/ms_payment': Operation not permitted
chown: changing ownership of '/var/lib/mysql/multi-master.info': Operation not permitted

If I run this container in the terminal before start docker desktop runs without any problem:

version: "3.8"
services:
  mariadb:
    image: mariadb
    container_name: mariadb
    environment:
      MYSQL_ROOT_PASSWORD: pass
      MYSQL_DATABASE: mybb
      MYSQL_USER: mybb
      MYSQL_PASSWORD: pass
    networks:
      - web
    restart: on-failure
    ports:
      - 3306:3306
    # volumes:
      # - ./mariadb:/var/lib/mysql
networks:
  web:
    external: true

For run it I just use type docker compose up -d in both cases.

I just encountered the same issue but for MySQL.

Previously using docker engine, the MySQL user was 999. Turns out under Docker Desktop this user is now 100998 for some reason...

Using chown recursively to set the owner to 100998 fixed it for me. (It should still be 999 though!)

p1-0tr commented 2 years ago

@muzzwood - that difference is expected. Long story short - we aim for DD not to require root privileges as much as possible, using ids that have meaning on the host would require elevated privileges. I explained this in a bit more detail here - https://github.com/docker/desktop-linux/issues/31#issuecomment-1161430097 .

muzzwood commented 2 years ago

Ah thanks for the explanation @p1-0tr, that makes sense. So this means I'll need to set the php-fpm user to be root in the docker file I guess 🤔

p1-0tr commented 2 years ago

@muzzwood - it depends what you are looking to achieve. If you want to use the same folder for bind-mounting with Docker Engine and DD interchangeably, I can't think of a good way to do that (without intervening chowns) off the top of my head. If you don't want the ownership of the shared files to change setting the user inside the container to root would work, because root inside the container maps to your local user on the host, so the files would remain in the ownership of your local user. But if you are looking to maintain easy access to those files on the host, you can also set a recursive ACL (https://linux.die.net/man/1/setfacl) on the shared folder, or create a group with id set to 100998 (groupadd -g 100998 <group name>) and add your user to it.

ARehmanMahi commented 2 years ago

It seems like a similar issue I had a few mins ago. Here's how I got it working. Let's say you have the following directory structure:

docker-compose.yml

version: '3.8'
services:
    php:
        image: php:8.1.8-fpm-alpine
        volumes:
            - ./:/var/www/html/

dockerize-project

docker-compose.yml app_files .... app_folders .... bootstrap storage ....

Now all the above files would map in /var/www/html/* i.e. /var/www/html/storage

And if you try to chown -R www-data storage, you will get permission denied error.

Solution Put the project in its own directory so that the web root looks like this /var/www/html/app/

docker-compose.yml

version: '3.8'
services:
    php:
        image: php:8.1.8-fpm-alpine
        volumes:
            - ./app/:/var/www/html/app/

dockerize-project

docker-compose.yml app

app_files .... app_folders .... bootstrap storage ....

And now chown -R www-data storage should work

docker-robott commented 1 year ago

There hasn't been any activity on this issue for a long time. If the problem is still relevant, mark the issue as fresh with a /remove-lifecycle stale comment. If not, this issue will be closed in 30 days.

Prevent issues from auto-closing with a /lifecycle frozen comment.

/lifecycle stale

ARehmanMahi commented 1 year ago

@p1-0tr I Just want to know if this issue was fixed.

Really appreciate all your replies on this issue and 1 of my other reported ones. I've been away from DD because I shuffle from system to system and such issues in DD were causing me delays. I Got notified of the closing by the bot so just curious.

cardonator commented 1 year ago

The issue is "fixed" in the sense that it is no longer completely broken. However, because of the design of Docker Desktop, it's not as pleasant of an experience as you would hope. The permissions end up in userland prefixed by 100. So if you chown to 999:999, the host permissions get changed to something like 100999:100999. I don't know that there is another option if they want Docker Desktop to be able to run under a user without root permissions to the system like native Docker.

ARehmanMahi commented 1 year ago

Thanks for the information, I'll give it a run next week. Yes, the UID & GID are less of a delight, but at least we are having less of a hassle over time. Appreciate.

docker-robot[bot] commented 1 year ago

There hasn't been any activity on this issue for a long time. If the problem is still relevant, mark the issue as fresh with a /remove-lifecycle stale comment. If not, this issue will be closed in 30 days.

Prevent issues from auto-closing with a /lifecycle frozen comment.

/lifecycle stale