search

Szum123321 / textile_backup

Backup Mod for Fabric
https://www.curseforge.com/minecraft/mc-mods/textile-backup
GNU General Public License v3.0
90 stars 30 forks source link

Fabric server crash (forced shutdown after 60 sec lag) during backup #12

Closed Zevac-ZSF closed 4 years ago

Zevac-ZSF commented 4 years ago

Hi there,

Since a couple of days my Fabric server (1.15.2) crashes during backup, strangely not every time, but the crash is due to a forced shutdown after >60 seconds of lag. This morning it was crashed again. And when I started it and logged in, a new backup started (as I have configured) and my cpu went straight to 100% on the server. After about 30 seconds I logged out again, and it went down to normal cpu usage. I waited untill the backup was finished and was able to log in again.

The strange thing is, I've been using your latest version since it came out (april 19th) and only lately this problem occurs, I've been updating the other used mods, Fabric, Java etc. No luck, my world is about 21 GB big, with a 14 GB .zip after backup, and what I have seen from the corrupted zips afterward is that their size is between 12 and 13 GB. But it is different everytime, and as said before also good backups inbetween.

I can imagine this is almost impossible to troubleshoot for you, but maybe you can add an setting that maximizes the cpu load, or amount of cores/threads used. That should prevent it, or do you run on the main game thread?

I hope you can help me, thank you in advance!

Zevac

Szum123321 commented 4 years ago

In 1.2.0 I've added multithreaded compression which, in case of zip, is always turned on and always uses all available threads. In older version backups were done just on a separate thread, which they still are.

It is entirely possible that in case of such a big file something inside minecraft server might time-out. Currently I don't have machine powerful enough to test that.

I think that the best solution would be to add a limit for how many cpu cores can be used for compression.

I will do it most likely today when only I finish my school stuff, so expect an update.

I hope this will help.

Szum.

Szum123321 commented 4 years ago

I think I repaired it. Please, I you could, test the 1.2.1 version and let me know if it works now. Filed are available at: https://github.com/Szum123321/textile_backup/releases/tag/1.2.1

Zevac-ZSF commented 4 years ago

Thanks, I'll test it right away, will let you know asap.

Zevac-ZSF commented 4 years ago

I set the compressionCoreCountLimit to 2 to test, after starting the server again and logging in I do see an error when the backup started and a short cpu spike causing a short lag. Is that cause by the start of the backup?

I've added a screenshot of the error in the attached file CaptureMCServerError

Szum123321 commented 4 years ago

The only possible repair i see here would be to update Apache Common Compress. Actually, as I look through release notes of newer versions I see quite a few bugs had been fixed. So updated version, 1.2.0B, is available in release.

Zevac-ZSF commented 4 years ago

Ok, I'm using 1.2.0b now, the login backup just finished, it still gives a one moment lag at startup of the backup (67 ticks behind), which still can be tricky with the flying machines I use. But after that it didn't create any lag, and did create a normal zip file again. So so far looking good. I'll see what happens when my 2nd-hourly backup kicks in.

Thanks so far!

Szum123321 commented 4 years ago

That's actually a great news! Please, keep me informed. If nothing new comes up in few hours I will publish updated files on curseforge. Those files will be called 1.2.1 (not B).

As for this 67 tick lag, before every backup sever gets saved, so for a short period of time it is possible that server pauses.

Thank you for reporting the bug!

Zevac-ZSF commented 4 years ago

Alright, just had a 2-hourly backup finished on 2 cores, and absolutely no problems, I saw 2 cores indeed going up, no lag issues, actually hardly any spike in the avg tick.

Looks like a perfect fix :-)

Szum123321 commented 4 years ago

Great! I will upload updates onto curseforge so where tommorwo.

Thanks for helping me fix that issue.