search

pterodactyl / panel

Pterodactyl® is a free, open-source game server management panel built with PHP, React, and Go. Designed with security in mind, Pterodactyl runs all game servers in isolated Docker containers while exposing a beautiful and intuitive UI to end users.
https://pterodactyl.io
Other
6.35k stars 1.55k forks source link

[Only affects wings 1.11.9 and up] Unable to mount an NFS "folder" inside the pterodactyl server volume. #5040

Open llek1000 opened 3 months ago

llek1000 commented 3 months ago

Current Behavior

After mounting an NFS share into a server's volume (for example /var/lib/pterodactyl/volumes/volume_id/data) and after starting the server, the server throws this error: Error Event [68d61a4e-0836-4694-99c0-6094e2819970]: failed to chown root server directory during pre-boot process: server/filesystem: chown: failed to chown during walk function: stat 0c89372c-9a08-4cee-b516-7dd0d4db3bca/part.602: file does not exist

I tried on two separate nodes and got the same result.

Expected Behavior

Mount an NFS share into a server's volume (for example /var/lib/pterodactyl/volumes/volume_id/data) start the server, the server will change its permissions and everything works fine - this was the behaviour before wings version 1.11.9

Steps to Reproduce

  1. Create a server - it does not matter which egg or docker image.
  2. Mount an NFS share into this servers volume
  3. Start the server

Panel Version

1.11.5

Wings Version

1.11.11

Games and/or Eggs Affected

No response

Docker Image

No response

Error Logs

https://ptero.co/igoselytov

Is there an existing issue for this?

ptoal commented 3 months ago

I can confirm this. I have an NFS4 mounted disk at /var/srv/gstorage1/wings1 that is owned by my pterodactyl:pterodactyl user. When attempting to start any server with 1.11.11, I get the same error. Reverting back to 1.11.8 works fine.

I haven't looked at the code yet, but my first guess is that wings is doing something with privileges that is changing its effective user id, and the NFS server is denying access.

llek1000 commented 3 months ago

Yes, I've also downgraded the node that runs the server with the mount. The node is private anyways so it does not really matter for now.

I do not think it's something to do with the user id, although I am not an expert by any stretch of imagination, so please excuse me if I'm wrong - since I do not really know how the internal processes of Wings work. However, I've spun up a new NFS4 share that allowed any user from the IP of the node to read/write and I got the same result.

From what I've seen in the log: WARN: [Mar 22 17:36:51.772] failed to determine root fs directory size error=server/filesystem: directorysize: failed to walk directory: walkdirat err: stat 0c89372c-9a08-4cee-b516-7dd0d4db3bca/part.602: file does not exist root=/var/lib/pterodactyl/volumes/a8c306f0-a28d-47c2-aeec-147b7c90df30 it cannot find the file 0c89372c-9a08-4cee-b516-7dd0d4db3bca/part.602 which is interesting, because the NFS4 share is mounted in 0c89372c-9a08-4cee-b516-7dd0d4db3bca/data/ and not in the root of the volume.

Also, Wings apparently has a problem with calculating the volume size: 

WARN: [Mar 22 17:36:51.772] failed to determine root fs directory size error=server/filesystem: directorysize: failed to walk directory: walkdirat err: stat 0c89372c-9a08-4cee-b516-7dd0d4db3bca/part.602: file does not exist root=/var/lib/pterodactyl/volumes/a8c306f0-a28d-47c2-aeec-147b7c90df30

Stacktrace:
stat 0c89372c-9a08-4cee-b516-7dd0d4db3bca/part.602: file does not exist
walkdirat err
github.com/pterodactyl/wings/server/filesystem.(*Filesystem).DirectorySize.func1
    github.com/pterodactyl/wings/server/filesystem/disk_space.go:170
github.com/pterodactyl/wings/internal/ufs.(*UnixFS).walkDir
    github.com/pterodactyl/wings/internal/ufs/walk_unix.go:59
github.com/pterodactyl/wings/internal/ufs.(*UnixFS).walkDir
    github.com/pterodactyl/wings/internal/ufs/walk_unix.go:76
github.com/pterodactyl/wings/internal/ufs.(*UnixFS).walkDir
    github.com/pterodactyl/wings/internal/ufs/walk_unix.go:76
github.com/pterodactyl/wings/internal/ufs.(*UnixFS).walkDir
    github.com/pterodactyl/wings/internal/ufs/walk_unix.go:76
github.com/pterodactyl/wings/internal/ufs.(*UnixFS).walkDir
    github.com/pterodactyl/wings/internal/ufs/walk_unix.go:76
github.com/pterodactyl/wings/internal/ufs.(*UnixFS).walkDir
    github.com/pterodactyl/wings/internal/ufs/walk_unix.go:76
github.com/pterodactyl/wings/internal/ufs.(*UnixFS).walkDir
    github.com/pterodactyl/wings/internal/ufs/walk_unix.go:76
github.com/pterodactyl/wings/internal/ufs.(*UnixFS).WalkDirat
    github.com/pterodactyl/wings/internal/ufs/walk_unix.go:33
github.com/pterodactyl/wings/server/filesystem.(*Filesystem).DirectorySize
    github.com/pterodactyl/wings/server/filesystem/disk_space.go:168
github.com/pterodactyl/wings/server/filesystem.(*Filesystem).updateCachedDiskUsage
    github.com/pterodactyl/wings/server/filesystem/disk_space.go:147
github.com/pterodactyl/wings/server/filesystem.(*Filesystem).DiskUsage
    github.com/pterodactyl/wings/server/filesystem/disk_space.go:113
github.com/pterodactyl/wings/server/filesystem.(*Filesystem).HasSpaceAvailable
    github.com/pterodactyl/wings/server/filesystem/disk_space.go:68
github.com/pterodactyl/wings/server/filesystem.(*Filesystem).HasSpaceErr
    github.com/pterodactyl/wings/server/filesystem/disk_space.go:53
github.com/pterodactyl/wings/server.(*Server).onBeforeStart
    github.com/pterodactyl/wings/server/power.go:194
github.com/pterodactyl/wings/server.(*Server).HandlePowerAction
    github.com/pterodactyl/wings/server/power.go:131
github.com/pterodactyl/wings/router/websocket.(*Handler).HandleInbound
    github.com/pterodactyl/wings/router/websocket/websocket.go:363
github.com/pterodactyl/wings/router.getServerWebsocket.func3
    github.com/pterodactyl/wings/router/router_server_ws.go:85
runtime.goexit
    runtime/asm_amd64.s:1650

server/filesystem: directorysize: failed to walk directory
parkervcp commented 3 months ago

This probably has to do with the fact that we are using the unix file stuff now as a change from the old filepath.

I am going to assume they are treated differently because it's externally mounted and the io package doesn't know how to handle the mount properly.

matthewpi commented 3 months ago

Where exactly on the filesystem are you mounting to?

I think I might know what the issue is.

llek1000 commented 3 months ago

Where exactly on the filesystem are you mounting to?

I think I might know what the issue is.

I am mounting exactly to /var/lib/pterodactyl/volumes/volume_id/data, of course, where "volume_id" is replaced by the actual volume_id.

Edit: Just in case it helps, I am using this mount under the minio egg, where I am mounting the data folder inside the pterodactyl volume to my NAS with NFSv4. Oh, and I do not have Squash enabled on my NFS server.

Hmmmz commented 3 months ago

I have a similar problem with this error that "file not exist" when checking file permission while starting on a NFS share with the new Wings version but I mount to "/var/lib/pterodactyl" instead, any ideas?

llek1000 commented 3 months ago

Hello, just checking in, any updates?

simonmicro commented 3 months ago

Hey, same problem here - just killed all my servers as I rolled out the update via Ansible. This also seemed to cause a lot of corrupt backups, idk how this came. Stupid mistake from my side.

DerMistkaefer commented 2 months ago

We have the same problem with servers/volumes that are created dynamically. We archive the volume dirs and then transfer them to the NFS share. The issue occurs sometime, so It's hard to analyze.

When I then restart wings everything works fine. So it seems there is a problem with new generated volume dirs. Like a cache or something.

So I think the problem is that when the server and the wings ufs filesystem instance is created that there is no volume dir exist. So then when we unzip the correct volume or for other the nfs mount get created there is another memory reference and the inital created ufs filesystem isn't exist anymore. So the link is broken. https://github.com/pterodactyl/wings/blob/develop/server/filesystem/filesystem.go#L41

I think the memory reference is the https://github.com/pterodactyl/wings/blob/develop/internal/ufs/fs_unix.go#L46 and need to be recreated if the basePath get changed. Or is not exist anymore.

I hope this helps, but I don't know the unix package and if is relevant if we have Openat2

DerMistkaefer commented 1 month ago

What worked for me, I don't delete the volume folder anymore. So I only delete the content of the volume folder, then unzip the files from the nfs share.

So the folder memory link don't get destroy.

azra026 commented 2 weeks ago

https://pastebin.com/CkcDH9hj

Still an issue with the latest release, only 1.11.8 works.

NobodyCentral commented 1 week ago

I'm also having this issue. I had wings version v1.11.11. I have an NFS share mounted to /mnt/ptero and changed the configs for wings to use that as its storage and when I try to start some but not all of my game servers it gets this error, but the moment I downgraded wings to version 1.11.8 it magically started working as normal.