Corrupted world on Windows 11

TeksuSiK commented 2 years ago

Describe the problem

After server restart, the server world files are corrupted. Everything works well while running pure Paper server without docker and also running same container on macOS or Linux. Recreating container doesn't help anything, after server restart I'm getting the same errors.

Container definition

version: "3.8"

services:
  mc:
    image: itzg/minecraft-server:java16
    ports:
      - "25565:25565"
    environment:
      TYPE: PAPER
      VERSION: 1.17.1
      EULA: "TRUE"
      ONLINE_MODE: "FALSE"
      COPY_CONFIG_DEST: /data
    volumes:
      - ./data/mc:/data
      - ./mc:/config

Container logs

Full server log is available here https://teksusik.pl/docker.log

itzg commented 2 years ago

I'm sorry to hear that happened. I'm afraid that is beyond the control of this image/container since ultimately it exec's to the requested server process. The logs provided do not show any of the container startup activity. Without those logs prior to the restart I don't know how you attempted to shutdown the server. Other external factors include the state of the attached ./data/mc host directory and any content augmented in from the ./mc host directory.

I would recommend discussing with the PaperMC team since it attempted but failed to recover the corrupted regionfile.

TeksuSiK commented 2 years ago

I'am restarting my server via stopping container from IntelliJ services tab, so the server is saving world, also there aren't any errors at this phase. I also probably know what's wrong. When I remove volume bindings from my docker-compose file and recreate container everything starts to work well.

itzg commented 2 years ago

Can this be closed now?

TeksuSiK commented 2 years ago

Well, I don't think this should work like this, but I don't know it's my fault or not, someone must try to reproduce it. In previous comment I meant that it only works without volume bindings and when I add them back it's again starts to serve errors.

itzg commented 2 years ago

I'm sorry I misunderstood your comment:

I also probably know what's wrong.

In any case, this is not a problem with the image. It's either something external or the PaperMC software itself. We can leave this issue open for others to find, but there's nothing I can do to fix it.

itzg commented 2 years ago

and just to make sure it wasn't a combination of those volume paths I used your compose file and tried a couple ways of restarting it in IntelliJ's service tab:

https://user-images.githubusercontent.com/988985/139678944-cfde1ee5-5832-4773-b37b-7a56e06ae8d5.mp4

TeksuSiK commented 2 years ago

Okay, so thanks for help. I will have to check what is wrong in my case.

Aniello12 commented 2 years ago

Same issue but only for minecraft version 1.18.1

docker-file `version: "3"

services: mc: image: itzg/minecraft-server:latest ports:

25565:25565 environment: EULA: "TRUE" VERSION: 1.18.1 TYPE: "BUKKIT" MEMORY: "10G" tty: true stdin_open: true restart: unless-stopped volumes:
./server:/data
attach a directory relative to`

itzg commented 2 years ago

@Aniello12 or @TeksuSiK are either of you running on Windows 11? I was chatting with someone yesterday with similar file corruption type issues and narrowed down that it was Windows 11 as the primary difference.

TeksuSiK commented 2 years ago

Yes, I'm working on Windows 11

MarkusKluge commented 2 years ago

I got the same issue here in Windows 11 but even with the default version (not paper). Whenever I chose a volume path for the container it will result in corrupted chunks.

As I haven't tested this on Windows 10 I can't say if this is an issue with W11.

How to "fix" it: When not using the volume parameter (default is docker generates a anonymous docker volume for the container) or using a named docker volume it will work and I can still access data with the windows explorer at \\wsl.localhost\docker-desktop-data\version-pack-data\community\docker\volumes\ without the chunk generation issues.

itzg commented 2 years ago

Thanks @MarkusKluge, your finding about the anonymous and named volume confirms another pattern I was starting to see. Also, thanks for the advice on how to access docker desktop file system. I never realized it was that easy even though I knew how to access regular distro file system.

itzg commented 2 years ago

To help others see it, I'm going to re-open this issue but change the title to mention Windows 11. I'll plan on submitting an issue with docker desktop; however, I would need those of you with Windows 11 to possibly help guide troubleshooting from there.

github-actions[bot] commented 2 years ago

This issue is stale because it has been open 30 days with no activity. Please add a comment describing the reason to keep this issue open.

AlbertoPimpo commented 2 years ago

After some trying on #1341 the approach of @MarkusKluge about using named volumes didn't work for me. I make it working by disabling the wsl backend and using hyper-v backend, so to me this is the only working work-around. @itzg maybe the problem is not docker but wsl itself, it can be an idea using a different distro as base distro for docker-desktop

itzg commented 2 years ago

@AlbertoPimpo I'm really surprised naked volumes didn't make difference.

Speaking of which, I thought I'd share this article I just saw that explains why host binding acts different than names volumes (in apparently not all cases)

https://www.docker.com/blog/file-sharing-with-docker-desktop/

Yes, it probably is a problem with WSL on Windows 11. I'm wondering if anybody has yet contacted Docker or Microsoft about this issue?

Zedifuu commented 2 years ago

I am experiencing this also with our server manager.

Docker Desktop on WSL2 On Windows 11, completely shreds the world on shutdown or restart :( On bare metal linux with Docker, it operates just fine :S

It's a peculiar one for sure, I wonder also if anyone has reached out to Docker or Microsoft.

AxelBrinck commented 2 years ago

I have this same problem on Ubuntu server. Like 30% of the chunks are misplaced. Only in 1.18.X, using any java 17 image. The error happens on all the server types, including vanilla.

I took the server to Windows 11 and only worked using named volumes.

Using named volumes could patch this issue. (Tested on W11)

Merith-TK commented 2 years ago

aight work around for now on windows if you 100% must have access to config files

1) in your mounts, put it in this order

"<volume name>:/data/world"
"./local/path/:/data" 2) start container 3) access container shell as root 4) as root run chown minecraft:minecraft /data -R 5) restart container

version: "3"

volumes:
  mc-world:

services:
  mc:
    image: itzg/minecraft-server:java17
    container_name: mc
    user: "1000:1000"
    stdin_open: true
    tty: true
    ports:
     - "25565:25565/tcp"
     - "25565:25565/udp" # for things that use UDP
    volumes:
      - "mc-world:/data/world" 
      - "./mc:/data"
    environment:
      PUID: "1000"
      PGID: "1000"
      TYPE: "fabric"
      VERSION: "1.18.1"
      PACKWIZ_URL: "https://gitlab.com/Merith-TK/vanillia-modpack/-/raw/server/server/pack.toml"
      EULA: "true"
      MEMORY: "3G"

the user and UID stuff is just to enforce the server running as a user and not as root, paranoid like that lol

NotEvenANeko commented 2 years ago

I disable the 'Use the WSL 2 based engine' option in Docker Desktop settings to use Hyper-V as backend, and it just works fine.

I think it's a bug of WSL 2.

quiquelhappy commented 2 years ago

has anyone found a solution to this project that doesnt require disabling WSL? This may be related? https://github.com/microsoft/WSL/issues/7977

What I'm seeing whenever a chunk is generated/loaded:

[13:22:17 ERROR]: Chunk [25, 10] header is truncated: expected 368640 but read 0

[13:22:17 ERROR]: [com.destroystokyo.paper.io.PaperFileIOThread] Failed to read chunk data for task: Task for world: 'world' at 26,9 poi: true, hash: 1394880741

java.lang.IllegalArgumentException: Negative position

at sun.nio.ch.FileChannelImpl.read(Unknown Source) ~[?:?]

at net.minecraft.world.level.chunk.storage.RegionFile.getChunkDataInputStream(RegionFile.java:604) ~[paper-1.18.2.jar:git-Paper-375]

at net.minecraft.world.level.chunk.storage.RegionFileStorage.read(RegionFileStorage.java:195) ~[?:?]

at net.minecraft.world.level.chunk.storage.RegionFileStorage.read(RegionFileStorage.java:187) ~[?:?]

at net.minecraft.world.entity.ai.village.poi.PoiManager.a(PoiManager.java:450) ~[?:?]

at net.minecraft.server.level.ServerLevel$1.readData(ServerLevel.java:327) ~[?:?]

at com.destroystokyo.paper.io.PaperFileIOThread$ChunkDataTask.run(PaperFileIOThread.java:508) ~[paper-1.18.2.jar:git-Paper-375]

at com.destroystokyo.paper.io.QueueExecutorThread.pollTasks(QueueExecutorThread.java:118) ~[paper-1.18.2.jar:git-Paper-375]

at com.destroystokyo.paper.io.QueueExecutorThread.run(QueueExecutorThread.java:51) ~[paper-1.18.2.jar:git-Paper-375]

[13:22:17 ERROR]: [com.destroystokyo.paper.io.PaperFileIOThread] Could not load chunk for task: Chunk task: class:com.destroystokyo.paper.io.chunk.ChunkLoadTask, for world 'world', (26,9), hashcode:1052169024, priority: -1, file IO thread has dumped the relevant exception above

here is a video showing the startup progress and the in-game experience: https://youtu.be/Sdo74YhdN6g

Here are all the logs for that same video: https://gist.github.com/quiquelhappy/a332d4a9eef5b46e6b81c544dd985651

itzg commented 2 years ago

@quiquelhappy there's workarounds mentioned in the comments above

quiquelhappy commented 2 years ago

@quiquelhappy there's workarounds mentioned in the comments above

Thanks, somehow I completely ignored one of the answers 🤔.

Hoping this gets fixed soon, seems like a WSL bug. Thanks for replying and for keeping up the repo! Have a nice day.

itzg commented 2 years ago

Luckily my PC doesn't have compatible hardware to run Windows 11 😉

Agreed, I'm surprised they haven't resolved this issue by now since it seems like quite a fundamental issue.

langestefan commented 1 year ago

@itzg not sure if this is still relevant, but I found that if I interacted with the docker commandline directly from my Ubuntu WSL2 machine I did not get any corrupted data. If I stopped a container using the windows command line I suddenly got issues.

itzg commented 1 year ago

I'm not sure if it is (I'm avoiding upgrading to Windows 11 because of things like this 😀), but thanks for the info.

langestefan commented 1 year ago

I think I got it. It's because the volumes are stored in the windows filesystem rather than the linux filesystem. When using a WSL2 backend with files stored in the linux filesystem there are no issues. Ps. windows 11 is awesome!

itzg commented 1 year ago

Thanks @langestefan for confirmation on both parts 😀

quiquelhappy commented 1 year ago

I think I got it. It's because the volumes are stored in the windows filesystem rather than the linux filesystem. When using a WSL2 backend with files stored in the linux filesystem there are no issues. Ps. windows 11 is awesome!

Pretty sure this was the case. For some reason, every time I installed Docker Desktop on my Windows computer, with a single WSL OS installed, it never automatically enabled Docker support for such OS. My recommendation to anyone having this issue is to use Docker from a WSL OS directly.

If you are new to Docker on Windows 11:

install Docker Desktop
if you get a prompt saying WSL isn't enabled on your system, you can enable it on the most recent versions of Windows 11 by looking for the 'WSL 2' app on the Microsoft Store; if that doesn't work, follow the guide linked by the prompt on the Docker Desktop installation
install any distro of your liking, for example Debian or Ubuntu; you can find these on the Microsoft Store
after installing and configuring your distro (simply search for the distro name and open it as you would do with any other application), go to Docker Desktop > Settings > Resources > WSL Integration, and enable the integration for the distro you just installed
when running commands, you can open your distro, or open the newly released Terminal app on Windows 11, making sure you select a docker-enabled distro when opening a tab.
you can find your files by going to /mnt/c/. If you want to be able to access the files from windows directly, make sure to use a volume pointing to a folder within /mnt/c/

Small note if you are transitioning to Windows 11 and are worried about your privacy (@itzg ✌️). There is no straight forward way of creating a local account while installing Windows 11 anymore, you'll have to enter a@a.com as your email when the setup asks you to login with your Microsoft Account, you should get an error and then be allowed to create a local account. When installing apps from the Microsoft Store, if you get a prompt asking you to login, just close it, the apps should install nevertheless. Happy holidays everyone!🎅

Verified this is working fine:

https://user-images.githubusercontent.com/12534357/209533668-6ea46ebf-fdf8-48d0-90e4-89d2ba0e941d.mp4

BiliBC034 commented 1 year ago

I've tried to do what @langestefan, @quiquelhappy and @Merith-TK said, but does not work.

@itzg not sure if this is still relevant, but I found that if I interacted with the docker commandline directly from my Ubuntu WSL2 machine I did not get any corrupted data. If I stopped a container using the windows command line I suddenly got issues.

Pretty sure this was the case. For some reason, every time I installed Docker Desktop on my Windows computer, with a single WSL OS installed, it never automatically enabled Docker support for such OS. My recommendation to anyone having this issue is to use Docker from a WSL OS directly...

aight work around for now on windows if you 100% must have access to config files

in your mounts, put it in this order

"<volume name>:/data/world"

"./local/path/:/data"

start container

access container shell as root

as root run chown minecraft:minecraft /data -R

restart container ...

I can not disable WSL and use Hyper-V as backend.

I've tried to mount the container in a named volume and worked, but I can not access there easily from win11. Is there any way to mount a named volume in a specific volume of win11 or to do a reference of the named volume into win11?

What I have now is this, but does not work:

version: "3"

services:
  mc:
    image: itzg/minecraft-server:latest
    container_name: mc-Final-Paradox
    ports:
      - 25565:25565
    expose:
      - 25565
    environment:
      EULA: "TRUE"
      DIFFICULTY: 'hard'
      VERSION: 1.16.5
      WHITE_LIST: "TRUE"
      GAMEMODE: survival
      MEMORY: '8G'
    tty: true
    stdin_open: true
    restart: unless-stopped
    volumes:
      - FP-data:/data
volumes:
  FP-data:
    driver: local
    driver_opts:
       o: bind
       type: none
       device: C:\Users\user\Desktop\minecraft_server\mincreft-data

If I leave it like this, works well, but I want to access the volume from win11:


...
volumes:
      - FP-data:/data
volumes:
  FP-data:

itzg commented 1 year ago

On Windows 10 at least, the managed volumes are located in \\wsl$\docker-desktop-data\data\docker\volumes

BiliBC034 commented 1 year ago

I found it, but I want to have the volume in the desktop, not as a direct access.

qruet commented 1 year ago

Coming from https://github.com/itzg/docker-minecraft-server/issues/1247. Ran into this exact issue when setting up a Ubuntu container environment for my server on Windows 11. Suggested solutions didn't work for me, found a workaround that may present as a new alternative for some. Utilizing the --world-container flag when starting up the server addresses incompatibility issues between NTFS and ext4 filesystems by introducing a separate Docker volume for storing worlds. Worlds are stored in a named Docker volume, the path of which is specified by the argument pointed out earlier, while the server directory remains a shared mapped volume with the host's filesystem.

itzg commented 1 year ago

@qruet can you point to documentation for --world-container?

qruet commented 1 year ago

@qruet can you point to documentation for --world-container?

The best documentation I could dig up can be found here. Note it's described as an optional setting for the bukkit.yml, but also exists as a startup flag. Mostly derived this option from the context brought up in random forum discussions when researching it.

It should be noted that bukkit no longer seems to support the --world-container option within the bukkit.yml, needs to be added as a startup flag.

BiliBC034 commented 1 year ago

For me, after a few months and some updates later, the problem is solved and everything works as it does when I had Win 10

PottedRosePetal commented 6 months ago

The issue very much still exists. I used some direwolf20 1.20 server on win11 and in the void dimension some chunks got deleted (so no big harm) and I seem to be unable to create new worlds using rftools dimensions. The logs are just a spam of the whole "failed to read chunk" error stuff (as can be found in this issue since the link above doesnt work anymore: https://github.com/itzg/docker-minecraft-server/issues/1247 ). The issue for microsoft mentioned above (https://github.com/microsoft/WSL/issues/7977) seems to have been closed due to inactivity, so probably no dice of them fixing that. Maybe the bug will disappear with windows 12...

just as a little update, I will probably not bother to try and fix it and just try to run this whole thing either without docker or on a linux system without windows :)

itzg / docker-minecraft-server