Backup purging hangs when using uncompressed differentials / incrementals

[Jadan1213]

I've noticed a few times that i have to manually re-enable chunk saving after a backup. i have no other mods that manipulate chunk saving installed.

I'm on forge 47.2.17, MC 1.20.1, mod ver. 3.4. I have Minecraft running as a local server on another machine, and my wife and I connect to it from our client machines.

I'm not 100% sure if it only happens when a save is done out of game (disconnected), but i feel like it is. I've noticed usually after logging back in after being off the server a while (no players) that if a backup was made, i have to manually re-enable saving with /save-on. This doesn't seem to happen when actually connected to the server, but i'm going to try and keep an eye out and see.

[MommyHeather]

Thanks for the report!

First up, do you have backups set to require activity?

Secondly, are you able to tell if your backup cycle continues as normal after this?

[MommyHeather]

Ah, also - immediately after you next encounter this, could you please retrieve a latest.log?

[Jadan1213]

i have backups set to run with player activity.. They seem to run fine after logging in and playing for a bit, i haven't noticed anything else. but i'll keep an eye out. the logs don't show anything except

[22:32:22] [Server thread/WARN]:

---

SAVING DISABLED - PREPARING FOR BACKUP!

---

[22:32:23] [Server thread/WARN]:

---

SAVE COMPLETE - PREPARING FOR BACKUP!

---

[22:32:23] [AB Active Backup Thread/INFO]: Preparing differential backup name: backup_2024-03-11_22-32-23

[Jadan1213]

it does seem to do a backup still after we log out, but i don't notice it doing further backups until after we log back in again

[MommyHeather]

i have backups set to run with player activity.. They seem to run fine after logging in and playing for a bit, i haven't noticed anything else. but i'll keep an eye out. the logs don't show anything except

[22:32:22] [Server thread/WARN]:

SAVING DISABLED - PREPARING FOR BACKUP!

[22:32:23] [Server thread/WARN]:

SAVE COMPLETE - PREPARING FOR BACKUP!

[22:32:23] [AB Active Backup Thread/INFO]: Preparing differential backup name: backup_2024-03-11_22-32-23

Are you able to provide the entire log please? There should be info after that, whether its an error or the complete message..?

[MommyHeather]

Only making one backup afterward logging off is indeed correct if you have player activity required, so that's not really a potential investigation route.

[Jadan1213]

i just rebooted my server after removing a mod, so as soon as it does it again i'll post the latest.log

[Jadan1213]

this is the latest.log from a couple reboots ago.

https://gist.github.com/Jadan1213/2b582c51f40ba46e8f4055e94f2092c5

[Jadan1213]

here's the one before that. (i'm not sure how long saving was off... still looking...

https://gist.github.com/Jadan1213/5290103cf77db582ee45e5b5041815dd

[MommyHeather]

Pausing server onPlayerLogout at 40827476, 551547 This, from my knowledge, isn't a vanilla feature.. I think I might know the cause now, but we shall see.

Are you playing a premade pack, or did you install all those mods manually? I might need to create an identical environment if I can't reproduce this manually.

[Jadan1213]

it's a mod that pauses season progression when players aren't logged in. it's called ready player fun

[Jadan1213]

the server still runs, it doesn't actually stop

[Jadan1213]

i have advanced backups set to use system time and not game time

[Jadan1213]

i'm playing on my own modpack that i put together. If you have modrinth i can send you an exported pack file that you could import into modrinth.

[Jadan1213]

i'm not sure if the issue would happen in a single player world or not. since i have it running on a dedicated server. I could zip up the server folder without the region files (so it's not huge lol) if you want

[MommyHeather]

Hmm. After some testing, I've been unable to reproduce it indev with readyplayerfun.

As such, yes I'd like to use a modrinth export, and a copy of your server (world folder not needed), thank you in advance!

[Jadan1213]

ok i ran around for a bit. logged out, and waited. it did the save with players logged out. and when i logged back in, i had to re-enable saving. Here's the latest.log

https://gist.github.com/Jadan1213/c0ad8de3c93d5d309e95c6249780dad3

[Jadan1213]

ok give me a little bit to get those files together. my wife is currently playing on the server lol.

[Jadan1213]

do you want me to include all the logs folders?

[MommyHeather]

No major need for a logs folder, no. I think the logs you've provided will give all the info the logs folder can, aside from perhaps a debug log.

[Jadan1213]

well if you know how to enable the debug log on server, please tell me lol. the server doesn't output one and i'm not sure how to turn it on!

[MommyHeather]

There should be a debug.log file in the logs folder - but if there isn't, just don't worry about it.

[Jadan1213]

there is not. i've been trying to figure out how to enable it to try and figure out a few other things with some mods. =(

[Jadan1213]

Here's the modrinth export(152.7MB): https://drive.google.com/file/d/1-0w49fkJlyoqjOQFIYSL4ld_rKgu0pam/view?usp=drive_link Here's the server directory sans world folder(945.2MB): https://drive.google.com/file/d/1BwIt_0IalhHwwM91mzTQuzQsjiGVKfE-/view?usp=drive_link

The server is set to use xms & xmx of 12GB, i don't know how much ram you have available, you might need to alter that. The client should use whatever your modrinth default ram limit is. I use 6GB and it runs okay.

You might have to disable the StayTrue texture pack, i know it gives my wife's pc some issues loading sometimes. so if you run into problems loading the client, that could help.

The files are fairly large as I have a "few" mods installed lol...

[Jadan1213]

just completed a backup while in game. it doesn't show in the server log output that the backup was completed, and it did not re-enable saving.

[Jadan1213]

i'm wondering if there may be an issue with the space checking for rotating backups. my server is running in a docker container with a folder mounted from my NAS to hold the backups, and i don't think that it's able to detect storage properly. Could this be causing an issue if it's attempting to enumerate the storage and delete old backups? maybe causing it to error out or cause it to not be able to complete its tasks prior to re-enabling saving?

[Jadan1213]

when i get chance i'll disable the deletion of old backups and see if it changes anything.

[MommyHeather]

Interesting.. That's a good thought. I've been unable to reproduce it with the server you sent so far, though I've not even managed to boot the client yet.

Those checks aren't ran if you run a snapshot with /backup snapshot, so that could be a good point to investigate.

[Jadan1213]

i let it run on its own, i didn't do a manual backup

[MommyHeather]

Yeah, both automatic and manual backups will run the checks, so if you're right in thinking it could be getting stuck on those, a manual snapshot would be a way to test that without disabling the checks.

Of course, the other option is setting all three config values to 0, which disables the checks outright.

[Jadan1213]

i did a /backup snapshot and it gave me an error. looks like the bakcup is hanging.

"Cannot start a snapshot whilst a backup is already running"

[Jadan1213]

it did backup the files though, so that's good at least

[MommyHeather]

Ah, that's not a surprise I guess. I'm not 100% sure you can break out of that without a server restart, but you can try /backup cancel, waiting a short while then running the snapshot again.

[MommyHeather]

and indeed, I can't reproduce it still.. Server is running on my pc using my NAS as storage, which is the slowest storage I have available - that's made no difference, and nothing I do on my client makes any difference.

[Jadan1213]

so i just did a /backup snapshot and it re-enabled saving. I've set the space check to 0, so i'll try a normal backup and see what happens

[Jadan1213]

It has to be something related to the fact that my NAS storage is being mounted into a docker container. I feel like it's not able to enumerate space usage and it's hanging. Since you can't reproduce it, i feel like it's an infrastructure thing and not a server configuration/mod thing.

[Jadan1213]

That has to be exactly it. After setting space purging to 0, the standard /backup start completed and re-enabled saving. You said that a /backup start would do space checks so i think that's it. Given that my use case is pretty niche (but possibly an issue others could run into) maybe adding a comment in the config that says something like "If you are mounting your backup storage into a docker container, space based purging should be set to 0 to prevent a possible issue with the backup hanging and not re-enabling chunk saving"

[Jadan1213]

unless you happen to have any ideas on how to prevent the hang lol. You're the coding master, not me! lol

[MommyHeather]

Hmm. I'd need to get a debugger attached to a server running inside a docker container to actually figure out a fix.

However, I can potentially implement a detection system for the time being...?

[Jadan1213]

well i happen to have a server running in a docker container and am willing to help if you want me to install/run anything on my server.

[MommyHeather]

I'll get you a jar to test with, though keep the old one handy.

[Jadan1213]

no problem. just let me know what to do =D

[MommyHeather]

AdvancedBackups-forge-1.20-3.4-dockercheck.zip

Give this one a test please - you'll have to disable the old jar. It should be able to check if a server is running inside docker, and if so, skip the purge checks.

[Jadan1213]

it gave me a 404 error

[MommyHeather]

Thanks github...

AdvancedBackups-forge-1.20-3.4-dockercheck.zip

I guess I'll try reuploading it? If not then I'll have to use something else to upload the file, strange. I've not had issues files to github like that before.

[MommyHeather]

yeah, that's still a 404

[Jadan1213]

if you want to send it to me on discord or use a google drive link, those work

[MommyHeather]

Oh, that's a thought. Rather than dming you, I can just send a link to a file uploaded to discord haha

AdvancedBackups-forge-1.20-3.4-dockercheck.zip

That link will work for 24 hours.

[Jadan1213]

yeah my server did a backup with no players and it completed perfectly. this is definitely the problem. downloaded the file.

[Jadan1213]

ok i reloaded the config after setting purge size to 100, and did a /backup reload-config running /backup start now