Unable to mount volume with s3fs plugin

marcelo-ochoa / docker-volume-plugins

Managed docker volume plugins

Eclipse Public License 2.0

88 stars 17 forks source link

Unable to mount volume with s3fs plugin #11

Open johny-mnemonic opened 2 years ago

johny-mnemonic commented 2 years ago

I am trying to add persistent storage to my RPi Swarm cluster and first wanted to start with MinIO as it is more modern than other options. I am unable to even mount the created volume though.

Steps to reproduce the behavior:

Install the plugin with docker plugin install --alias minio mochoa/s3fs-volume-plugin-armv7l --grant-all-permissions --disable DEFAULT_S3FSOPTS='allow_other,url=http://swarm:9002,use_path_request_style,nomultipart'
Set access/secret keys
Enable the plugin
Create volume with docker volume create -d minio octoprint

Got error while trying to mount the volume with


docker run --rm -it --volume-driver minio -v octoprint:/mnt arm32v7/alpine:3.12 /bin/sh
docker: Error response from daemon: VolumeDriver.Mount: error mounting octoprint:
df: /var/lib/docker-volumes/fc6cb3928d55a6a14b7280e768285e888351c91996c1aa464c7d0d2d7a584361: Software caused connection abort

fstype should be s3fs.

6. No bucket created in MinIO (according to s3 ls).
7. Created manually by `mc mb minio/octoprint`
8. Now the error is different:

docker run --rm -it --volume-driver minio -v octoprint:/mnt arm32v7/alpine:3.12 /bin/sh docker: Error response from daemon: failed to chmod on /var/lib/docker/plugins/5c1d484c3d37486708844217a3d7f230e27d1b2e52285f696fcc3f9d69248e98/propagated-mount/c6d472d807b5e4c0f6c0a34480e20682f97bdc399c9c62e03e6e3e4ca1097476: chmod /var/lib/docker/plugins/5c1d484c3d37486708844217a3d7f230e27d1b2e52285f696fcc3f9d69248e98/propagated-mount/c6d472d807b5e4c0f6c0a34480e20682f97bdc399c9c62e03e6e3e4ca1097476: input/output error.



**Server (please complete the following information):**
 - OS: Raspbian
 - Version: buster
 -  Arch: armv7

**Additional context**
I have tried many s3fs options combinations, but nothing worked. When I removed `use_path_request_style` or added `uid=0,gid=0` (to prevent chown) it failed the same as when the bucket didn't exist in MinIO.
I tried searching around a bit and seems like it is incorrect to try to do chown on S3 as it is not POSIX filesystem. Not sure why the plugin does it, or whether it is the plugin or docker itself. Not sure how other S3 compatible filesystems deal with this, but MinIO obviously can't handle it...

NightmareZero commented 1 year ago

To get more information about what happened during the mount process, you can use the command journalctl -ru docker | more. This will show you a log of events, and you can search for any errors or warnings that might explain the issue. In my case, I found out that my Minio address was configured incorrectly thanks to this command.

NightmareZero commented 1 year ago

During my testing, I discovered that it's not possible to create a bucket and subdirectories automatically. This means that you'll need to create them manually before you can start using them.

maretodoric commented 1 year ago

I'm having somewhat similar issue but it's tied to specific images. For example, this issue occurrs when i try to use adguard/adguardhome image to mount s3fs to /opt/adguardhome/conf and /opt/adguardhome/work directories.

For some reason, /opt/adguardhome/conf mounts fine, but not /opt/adguardhome/work...

Below is an example, adguard-conf and adguard-work are volumes/buckets of s3fs plugin.

This works

docker run --rm -it -v adguard-conf:/opt/adguardhome/conf -v adguard-work:/opt/adguardhome/worky adguard/adguardhome

This works

docker run --rm -it -v adguard-conf:/opt/adguardhome/conf adguard/adguardhome

This does NOT work

docker run --rm -it -v adguard-conf:/opt/adguardhome/conf -v adguard-work:/opt/adguardhome/work adguard/adguardhome
docker: Error response from daemon: failed to chmod on /var/lib/docker/plugins/f7d11b8c34f653d03fabc15867fe46ee243618a296451f7c5a4be0c4944dd8cf/propagated-mount/4ee850eaef3fbee9532faac07de260313ce3b2a81f5bbd76d3bb9524eeb54f21: chmod /var/lib/docker/plugins/f7d11b8c34f653d03fabc15867fe46ee243618a296451f7c5a4be0c4944dd8cf/propagated-mount/4ee850eaef3fbee9532faac07de260313ce3b2a81f5bbd76d3bb9524eeb54f21: input/output error.
See 'docker run --help'.

I've created container with "normal" (local) volume driver, checked both directories and problematic /opt/adguardhome/work directory contains subdirectory data. So i thought about creating volumes with nonempty and still same issue. I've also checked, both directories, work and conf have ownership of nobody (uid:gid according to stat is 65534:65534) so I've also added uid=65534,gid=65534 to s3fsopts and i still have the same issue.

Cannot figure it out why..

harenber commented 11 months ago

See the same what @maretodoric reports in https://github.com/marcelo-ochoa/docker-volume-plugins/issues/11#issuecomment-1715367569 when using a simple, empty bucket on a Wasabi backend and try to mount it with alpine:

harenberg@compass20:~$ docker volume create -d s3fs s3fsbucket
s3fsbucket
harenberg@compass20:~$ docker run --rm -it -v s3fsbucket:/mnt alpine
docker: Error response from daemon: failed to chmod on /var/lib/docker/plugins/1171ee995bd62d1a5d2bed7cdb8223fb81f22af944de1604624da263e9ebe04a/propagated-mount/6faa86eb083fa01635621f30f083f831db2126a83536ab45e31413761c443eee: chmod /var/lib/docker/plugins/1171ee995bd62d1a5d2bed7cdb8223fb81f22af944de1604624da263e9ebe04a/propagated-mount/6faa86eb083fa01635621f30f083f831db2126a83536ab45e31413761c443eee: input/output error.
See 'docker run --help'.
harenberg@compass20:~$

journalctl shows:

Dec 04 16:12:20 compass20 dockerd[2135145]: time="2023-12-04T16:12:20.028398815+01:00" level=error msg="Handler for POST /v1.43/containers/create returned error: failed to chmod on /va
r/lib/docker/plugins/1171ee995bd62d1a5d2bed7cdb8223fb81f22af944de1604624da263e9ebe04a/propagated-mount/6faa86eb083fa01635621f30f083f831db2126a83536ab45e31413761c443eee: chmod /var/lib/
docker/plugins/1171ee995bd62d1a5d2bed7cdb8223fb81f22af944de1604624da263e9ebe04a/propagated-mount/6faa86eb083fa01635621f30f083f831db2126a83536ab45e31413761c443eee: input/output error"
Dec 04 16:12:20 compass20 dockerd[2135145]: time="2023-12-04T16:12:20+01:00" level=error msg="2023/12/04 15:12:20 Entering go-plugins-helpers unmountPath" plugin=1171ee995bd62d1a5d2bed
7cdb8223fb81f22af944de1604624da263e9ebe04a
Dec 04 16:12:20 compass20 dockerd[2135145]: time="2023-12-04T16:12:20+01:00" level=error msg="2023/12/04 15:12:20 Entering go-plugins-helpers capabilitiesPath" plugin=1171ee995bd62d1a5
d2bed7cdb8223fb81f22af944de1604624da263e9ebe04a
Dec 04 16:12:20 compass20 dockerd[2135145]: time="2023-12-04T16:12:20+01:00" level=error msg="2023/12/04 15:12:20 Entering go-plugins-helpers getPath" plugin=1171ee995bd62d1a5d2bed7cdb
8223fb81f22af944de1604624da263e9ebe04a
Dec 04 16:12:20 compass20 dockerd[2135145]: time="2023-12-04T16:12:20+01:00" level=error msg="2023/12/04 15:12:20 Entering go-plugins-helpers capabilitiesPath" plugin=1171ee995bd62d1a5
d2bed7cdb8223fb81f22af944de1604624da263e9ebe04a
Dec 04 16:12:19 compass20 dockerd[2135145]: time="2023-12-04T16:12:19+01:00" level=error msg="2023/12/04 15:12:19 [/var/lib/docker-volumes/6faa86eb083fa01635621f30f083f831db2126a83536a
b45e31413761c443eee -o nomultipart,allow_other,use_path_request_style,url=https://s3.eu-central-2.wasabisys.com/,bucket=s3fsbucket]" plugin=1171ee995bd62d1a5d2bed7cdb8223fb81f22af944de
1604624da263e9ebe04a

BarryLuijten commented 11 months ago

I found a post that describes how to use the s3fs package (the package used by this plugin) with Minio. Apparently it's required to add the driver option use_path_request_style. This worked for me at least.

So my command to configure the docker plugin is:

docker plugin set s3fs DEFAULT_S3FSOPTS="url=https://<my-minio-url>,endpoint=<my-minio-region>,use_path_request_style"

I also found a description of all driver options, in case you want to play around.

NightmareZero commented 11 months ago

Have you tried manual mounting? I have looked at the code of this plugin, and the principle is to use s3fs-fuse, your reserved username, password, and address to mount it to a specific directory

See the same what @maretodoric reports in #11 (comment) when using a simple, empty bucket on a Wasabi backend and try to mount it with alpine:

harenberg@compass20:~$ docker volume create -d s3fs s3fsbucket
s3fsbucket
harenberg@compass20:~$ docker run --rm -it -v s3fsbucket:/mnt alpine
docker: Error response from daemon: failed to chmod on /var/lib/docker/plugins/1171ee995bd62d1a5d2bed7cdb8223fb81f22af944de1604624da263e9ebe04a/propagated-mount/6faa86eb083fa01635621f30f083f831db2126a83536ab45e31413761c443eee: chmod /var/lib/docker/plugins/1171ee995bd62d1a5d2bed7cdb8223fb81f22af944de1604624da263e9ebe04a/propagated-mount/6faa86eb083fa01635621f30f083f831db2126a83536ab45e31413761c443eee: input/output error.
See 'docker run --help'.
harenberg@compass20:~$

journalctl shows:

Dec 04 16:12:20 compass20 dockerd[2135145]: time="2023-12-04T16:12:20.028398815+01:00" level=error msg="Handler for POST /v1.43/containers/create returned error: failed to chmod on /va
r/lib/docker/plugins/1171ee995bd62d1a5d2bed7cdb8223fb81f22af944de1604624da263e9ebe04a/propagated-mount/6faa86eb083fa01635621f30f083f831db2126a83536ab45e31413761c443eee: chmod /var/lib/
docker/plugins/1171ee995bd62d1a5d2bed7cdb8223fb81f22af944de1604624da263e9ebe04a/propagated-mount/6faa86eb083fa01635621f30f083f831db2126a83536ab45e31413761c443eee: input/output error"
Dec 04 16:12:20 compass20 dockerd[2135145]: time="2023-12-04T16:12:20+01:00" level=error msg="2023/12/04 15:12:20 Entering go-plugins-helpers unmountPath" plugin=1171ee995bd62d1a5d2bed
7cdb8223fb81f22af944de1604624da263e9ebe04a
Dec 04 16:12:20 compass20 dockerd[2135145]: time="2023-12-04T16:12:20+01:00" level=error msg="2023/12/04 15:12:20 Entering go-plugins-helpers capabilitiesPath" plugin=1171ee995bd62d1a5
d2bed7cdb8223fb81f22af944de1604624da263e9ebe04a
Dec 04 16:12:20 compass20 dockerd[2135145]: time="2023-12-04T16:12:20+01:00" level=error msg="2023/12/04 15:12:20 Entering go-plugins-helpers getPath" plugin=1171ee995bd62d1a5d2bed7cdb
8223fb81f22af944de1604624da263e9ebe04a
Dec 04 16:12:20 compass20 dockerd[2135145]: time="2023-12-04T16:12:20+01:00" level=error msg="2023/12/04 15:12:20 Entering go-plugins-helpers capabilitiesPath" plugin=1171ee995bd62d1a5
d2bed7cdb8223fb81f22af944de1604624da263e9ebe04a
Dec 04 16:12:19 compass20 dockerd[2135145]: time="2023-12-04T16:12:19+01:00" level=error msg="2023/12/04 15:12:19 [/var/lib/docker-volumes/6faa86eb083fa01635621f30f083f831db2126a83536a
b45e31413761c443eee -o nomultipart,allow_other,use_path_request_style,url=https://s3.eu-central-2.wasabisys.com/,bucket=s3fsbucket]" plugin=1171ee995bd62d1a5d2bed7cdb8223fb81f22af944de
1604624da263e9ebe04a