New Install - init-stage2 failed - changing ownership not permitted on logs folder

[SaschaHenning]

Checklist

• Have you pulled and found the error with jc21/nginx-proxy-manager:latest docker image?
  • Yes
• Are you sure you're not using someone else's docker image?
  • Yes
• Have you searched for similar issues (both open and closed)?
  • Yes

Describe the bug After an update of the docker image the init did fail (see below). So I created a new container.

1. start container
2. "logs" folder in "data" is created successfully
3. error as shown here in the log:

[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] 01_perms.sh: executing... 
Changing ownership of /data/logs to 0:0
chown: changing ownership of '/data/logs': Operation not permitted
[cont-init.d] 01_perms.sh: exited 1.
!!!!!
 init-stage2 failed.
!!!!!

Nginx Proxy Manager Version 2.9.5 - pulled jc21/nginx-proxy-manager:latest today

Operating System Portainer LXC on Proxmox server (x64) Maybe important detail: Mapped folders for data etc. are on an NFS Share.

[bynicolas]

Still same issue in 2.9.12 Also using data folder on a NFS share

[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] 01_perms.sh: executing... 
Changing ownership of /data/logs to 0:0
chown: changing ownership of '/data/logs/proxy_host-15.log': Operation not permitted
chown: changing ownership of '/data/logs/error.log': Operation not permitted
chown: changing ownership of '/data/logs/default.log': Operation not permitted
chown: changing ownership of '/data/logs/proxy_host-1.log': Operation not permitted
chown: changing ownership of '/data/logs/letsencrypt-requests.log': Operation not permitted
chown: changing ownership of '/data/logs/proxy_host-2.log': Operation not permitted
chown: changing ownership of '/data/logs/default_host.log': Operation not permitted
chown: changing ownership of '/data/logs/proxy_host-3.log': Operation not permitted
chown: changing ownership of '/data/logs/proxy_host-4.log': Operation not permitted
chown: changing ownership of '/data/logs/proxy_host-5.log': Operation not permitted
chown: changing ownership of '/data/logs/proxy_host-6.log': Operation not permitted
chown: changing ownership of '/data/logs/proxy_host-7.log': Operation not permitted
chown: changing ownership of '/data/logs/proxy_host-8.log': Operation not permitted
chown: changing ownership of '/data/logs/proxy_host-9.log': Operation not permitted
chown: changing ownership of '/data/logs/proxy_host-10.log': Operation not permitted
chown: changing ownership of '/data/logs/proxy_host-11.log': Operation not permitted
chown: changing ownership of '/data/logs/proxy_host-14.log': Operation not permitted
chown: changing ownership of '/data/logs/proxy_host-12.log': Operation not permitted
chown: changing ownership of '/data/logs/proxy_host-13.log': Operation not permitted
chown: changing ownership of '/data/logs/proxy_host-16.log': Operation not permitted
chown: changing ownership of '/data/logs/proxy_host-17.log': Operation not permitted
chown: changing ownership of '/data/logs': Operation not permitted
[cont-init.d] 01_perms.sh: exited 1.
!!!!!
 init-stage2 failed.
!!!!!

[the1ts]

The chown is being done as the root user, does the NFS export have no_root_squash? If not then the NFS server is simply reducing the permissions of the remote root user stopping docker and its containers from doing a chown 0:0.

[SrFrancia]

I'm having the same issue running Rocky Linux 8.5 and using podman 3.4.2 on a proxmox VM just checked my filesystem and its XFS does it have a similar system to root_squash @the1ts (I just learnt about it)? This is what happens when running from scratch just after everything is downloaded

Creating nproxym_nproxym_1 ... done
Attaching to nproxym_nproxym_1
nproxym_1  | [s6-init] making user provided files available at /var/run/s6/etc...exited 0.
nproxym_1  | [s6-init] ensuring user provided files have correct perms...exited 0.
nproxym_1  | [fix-attrs.d] applying ownership & permissions fixes...
nproxym_1  | [fix-attrs.d] done.
nproxym_1  | [cont-init.d] executing container initialization scripts...
nproxym_1  | [cont-init.d] 01_perms.sh: executing...
nproxym_1  | Changing ownership of /data/logs to 0:0
nproxym_1  | chown: cannot read directory '/data/logs': Permission denied
nproxym_1  | [cont-init.d] 01_perms.sh: exited 1.
nproxym_1  |
nproxym_1  | !!!!!
nproxym_1  |  init-stage2 failed.
nproxym_1  | !!!!!

Nginx Proxy Manager Version 2.9.16 - pulled jc21/nginx-proxy-manager:latest today

[WaldHabets]

Try using the :Z suffix for your volumes, so

--volume /path/to/data:/data:Z

instead of

--volume /path/to/data:/data

[bynicolas]

Still had issues and adding :Z suffixe did not work for me.

Since only the logs directory seems to have it's ownership changed, I've simply relocated the volume mapping for that log directory locally on the host (and not on the NFS share)

so in other words, I've added --volume /path/to/local/nginx-proxy-manager/data/logs:/data/logs in addition to the previous --volume /path/to/nfs/nginx-proxy-manager/data:/data

This workaround works fine for me, but maybe you'd want to create a cronjob to rsync or move over your log files to the nfs share in your particular situation

[bynicolas]

@the1ts

The chown is being done as the root user, does the NFS export have no_root_squash? If not then the NFS server is simply reducing the permissions of the remote root user stopping docker and its containers from doing a chown 0:0.

I have my NFS share set to map all users to the admin user on my NFS server. It seems to me that the container shouldn't have permissions issues, but I don't understand why I'm still getting this error. For the record, I'm using Synology DSM as the NFS host.

If anyone else has a better explanation or a fix, I'd be glad to learn about it, meanwhile, the workaround above will do just fine for me.

[the1ts]

@bynicolas Yeah, I can cause this exact error by using your setup. I have the standard for a NFS only share. i.e. mapping none and security sys. This means that the client sets the permissions not the server. If you squash permissions with mapping a root user on the client doesn't have root on the server so cannot chown 0:0 and scripts in containers die.

[bynicolas]

@the1ts Thanks for the tip, I managed to get it working (kind of) with the default setup as well mapping none security sys

The only thing now is that I see a bunch of warnings about failed logrotate due to insecure permissions

...
[5/20/2022] [10:09:19 PM] [Setup    ] › ⚠  warning   Error: Command failed: logrotate /etc/logrotate.d/nginx-proxy-manager 
error: skipping "/data/logs/default-host_access.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
...

Do you happen to know how to how to fix this? Maybe I'm using incorrect mount options?

[the1ts]

All I can suggest is perhaps fix the permissions manually since they've been somewhat broken by the NFS issues in the past. My /data/logs is chown 0:0 and chmod 755 and /data/logs/*log is again chown 0:0 but chmod 644.

This looks correct as it follows the chmod for /data/logs in the NPM scripts and the 644 for the files implied by the logrotate config.

[jpharaoh27]

Hey all, I'm having a similar issue here. In the screenshot, I created and chown/chmod the /data/logs directory and I got the chown error. Before I created the logs directory, I was getting a "mkdir - permission denied" error when it was trying to make /data/logs.

Any advice/help will be super appreciated.. been racking my head on this for days lol

[github-actions[bot]]

Issue is now considered stale. If you want to keep it open, please comment :+1:

[ensleep]

I want to know why npm need chown?The root in npm has the right to edit file in nfs . Why It need a right to run chown?

[jc21]

because the image can run nginx as non-root, if specified with PUID env variable.

[ian351c]

[EDIT] So it looks like there's more chowning going on than just Stage 2. I created a PR that skips chowning for the /data and /etc/letsecrypt folders since those (for me) are mounted remotely (VMware HGFS) and don't allow chowning to happen. It's PR-3792 and implements a SKIP_FILE_OWNERSHIP environment variable that, if set to true, will skip chowning anything in /data or /etc/letsencrypt. This works for me because I I mounted the HGFS volume with the same UID/GID that I am running the NPM container as, so there is no need to attempt to chown those files. [END EDIT]

For folks that don't know how this works (which was me up until 10 minutes ago): @jc21 is using s6-overlay by Just Containers. This framework appears to automatically try to change ownership of (at least some of) the data/config/log files stored in volumes or binds for the container when PUID or GUID is set. This fails in certain situations (for example: I am running NPM in a Docker container on Debian VM inside VMWare Fusion on a Mac mini M1). To override this behavior, set the following environment variable: S6_BEHAVIOUR_IF_STAGE2_FAILS=0 in the docker command or compose file. Be aware that I have no idea what else this might break, but it seems to be working for me. I still have read/write access to all my files inside the container, I just can't change ownership.

References: NPM Dockerfile (see line 15) Customizing s6-overlay behavior

[ian351c]

@jc21 I'm not sure how to request this to be merged, but if you point me in the right direction, I'll go that way...