msmhq / msm

An init script for managing Minecraft servers
http://msmhq.com
GNU General Public License v3.0
1.22k stars 225 forks source link

Backups fail on large files #118

Open cameroncros opened 11 years ago

cameroncros commented 11 years ago

Disabling level saving... Done. Forcing save... Done. Synchronising worlds in RAM to disk... Done. Backing up the entire server directory... zip error: Out of memory (append_ulong_to_mem) Done.

This is the error i get when backing up my ~12gb minecraft server. As 90% of the server size is dynmap, it would be nice if it were possible to exclude plugins/dynmap (or any particular folder) from the backup. At best it should alert us that it failed.

There is enough hard disk on the server to successfully backup, but zip failed and msm didn't notice.

khoek commented 11 years ago

Hi cameroncors!

Your suggestion is very much achievable with little change to the current backup system. I however am no expert at shell/bash scripting (I am however a veteran java dev, but that's pretty much useless in this situation), so you are going to have to wait for someone like marcus to view this ticket and commit an actual patch.

marcuswhybrow commented 11 years ago

Thanks @cameroncros, I will most likely add msm-backup-exclude=plugins/dynmap to MSM's server.properties config settings.

cameroncros commented 11 years ago

Does the exclude feature already exist or are you adding it in the future? Will/is it possible to add multiple exclude directories/files.

marcuswhybrow commented 11 years ago

I will add that feature in the future. The value I will pas directly to the exclude flag of the zip command. So you will be able to exclude multiple files yes.

cameroncros commented 11 years ago

I have put in a pull request for a version of the code with an exclude option put in, however, it doesn't actually solve the issue for me, but I only have 400MB free memory with the server running and my server is ~1GB so that is likely the main cause. I don't know whether there is a work around for this, but at least this patch might help someone else. Interestingly, the error message I get now is: zip error: Out of memory (local_to_wide_string). I suspect this might be a limitation of zip/my server.

cameroncros commented 11 years ago

One suggestion I have, although i appreciate it may not be an easy change, is to use a different archiving and compression method that doesn't require a similar amount of free ram as the server disk size. I found that some zip implementations have a streaming option that makes them relatively low memory usage.

khoek commented 11 years ago

@cameroncros Perhaps gzip? It's awesome.

cameroncros commented 11 years ago

you mean using tar -zxvf [dir] output.tar.gz? That actually works on my server, and its a pretty decent idea, i will write a patch but im very doubtful that it will be merged as it might break other peoples stuff. Prehaps i will add a patch that includes the option for zip or tar. But im not going to do it anytime soon