Save-off on cartography

[mattrl]

There is the problem that if you save-off for cartography and the cartography takes an exceptionally large mount of time when save-on and save-all or first save take place the server will lag to hell and drop all players.. possibly even crash.

Obviously the smart thing to do would be to run cartography out of peak time where it is less likely to matter.

Making a copy of the world direction could complicate the configuration perhaps but it would be advantageous. I do this myself for overviewer runs. As rsync is not file locking there is no need to turn saves off, just to do a save-all to ensure that a save has taken place. Any corruption in the copy is irrelevant for the purpose of cartography / overviewer map production.

Just putting it out there as an idea.

[mattrl]

It is really only about 2 extra lines per job.. 4 in total.. maybe a few lines for variables in the configuration area. But whether users will understand or not or whether you think it is warranted.

[katanacrimson]

Best thing to do is to run your preferred carto script in between server restarts, tbh. Server shutdown, then carto, full backup, then start back up is how I do it.

[endofzero]

I've already started testing a form of this method with the biome section. I am going to look into rsync and start testing with that. Only thing I can really see as a potential issue is that you may have to keep an eye on your world size if space is short as it would be doubled.

[endofzero]

Any thoughts about having the rsync actions an option?

So if you don't require that amount of time to run scripts, you could just do it while the server waits for it to finish. Otherwise, you can enable it to have a 'mapper' copy...

[katanacrimson]

"Only thing I can really see as a potential issue is that you may have to keep an eye on your world size if space is short as it would be doubled."

Could always do a little bash magic to get the free space left, get the space that the world takes up, and warn if it'll leave only a certain amount of space left or less.

Might not be easy though, just an idea.

[demonspork]

how did I not notice this when I posted mine? there were only 2 listed for me when I saw this list and made my new post about the same issues and ideas. odd, but nevertheless.

running while the server is offline is a joke, because with a 100,000+ chunk map it takes a long time to do and I like to do it at least once an hour, meaning my server would be off for 15+ minutes every one hour.

I am also currently working on a way to use modified by dates on the chunks to determine which chunks have been edited and make a list of them for every overviewer run and to use --chunklist option with this list. This will speed up runs quite a bit because half of the time is spent scanning each chunk for changes. The list generated by the backup section is all changes since the last full backup, I only want changes since the last overviewer (usually about an hour).

We should definitely have a lock file for cartography and overviewer runs, because if they overlap when run by cron jobs the server would slow to a crawl and overviewer would even crash out from the errors of 2 instances running.

[mattrl]

I am using a nested IF statement checking the following two conditions:

ONLINE1=$(pgrep -f -c 'rsync -ap') ONLINE2=$(pgrep -f -c 'python /data/overviewer/gmap.py')

If false, proceed. Of course a lock file is probably better but I had to make do with what I could make myself.. what was possible with my skill level :)

At some point something appeared to break in script and I stopped getting accurate chunklists so I started using the following:

nice -n 15 ionice -c3 rsync -vap --stats /home/mcohyeah/minecraft/OCAU /data/worlddata/ | grep -v 'uptodate' > $CHUNKLIST

I 'think' this was because it requires -vv which I was testing as only -v because of the crontab problems with excessive output.

[demonspork]

Lock files are really easy

check for lockfile - you would want to add an option to override the lock file.

if [[ -e lockfile ]]; then echo "Previous run hasn't completed" fi

to create lockfile

touch lockfile

I am getting ready to add a fork and then maybe integrate parts of my revision into this - I am mainly worried about the amount of time between save-off and save-on with the minecraft-overviewer. If someone is running overviewer, the size of their map is probably not going to be a space issue for them considering that the cache and the image output of minecraft overviewer is going to exceed 10GB if the map is 100,000 chunks or more. So the extra space needed for a copy of the world is of little consequence in comparison.

Lock files are still on my to do list, but I have the biome extractor and the minecraft-overviewer (working on c10t now) using the world copy instead of the actual world, only requiring saving to be off long enough for the rsync to run.

[endofzero]

i've been testing with rsync... seems the way to go... i just wanted a day for cron jobs to push them before any commits

[endofzero]

Ok, I've addressed both issues in the latest version of my commit... I've added the lockfile test to the new 'sync' function, but should be really easy to add to overviewer and such as well... I just put it there since i was testing sync as well.

So the idea is that you would run 'minecraft.sh sync' to update your offline folder set by OFFLINE_NAME in the main config section.

Then you can run the mappers on that offline folder. I added a quick size comparision and removed the console messages for the mappers as they no longer affect anyone on the server.

[endofzero]

Alright... can someone test the latest commit of the master branch.... it should be using the offline folder for all mapper functions... all mapper functions have their own lockfiles that they look at.

If that looks ok, we can close this issue.

I use Debian squeeze and all of the functions are working perfectly on my system.

[endofzero]

Going to consider this working as expected...

[demonspork]

the minecraft.sh sync command seems unnecessary, it would make more sense as a function that is called every time something that would need it done runs, like sync_offline and could be called by the overviewer section, the cartography section and any other section of the code that would need it updated. the way it is now you have to add two commands any time you want to do one task. If I run minecraft.sh overviewer, there is no reason for me not to sync the directories first, so having to run it separately is pointless.

[endofzero]

I thought about doing it that way. I created the command to allow for more control of when the saving it turned off to account for the users. I have all my mapping done at the same time every day. It makes no sense to have it sync 6 times when it only needs to sync once. You sync, then you can issue all your mapping commands without affecting the users.

I suppose we could transition it into a function and allow something like 'minecraft.sh overviewer sync' to kick off a pre-sync before running overviewer... This should allow for some flexibility when issuing commands... Best of both worlds, if it is done right, it would seem...

Thoughts?

[demonspork]

that would actually be perfect, and not hard to pull off. Just move the code from the sync section into its own function, call the function from the minecraft.sh sync section simply insert a check to see if $2 says "sync" while running the overviewer or cartograph.

I didn't think of doing multiple operations at once, because I only ever use one, but if you are generating a daily image of the map for posterity with c10t and updating the overviewer next, you would really save effort by syncing only once for both of those tasks.

my bash fu is pretty novice, so I will leave checking that stuff to you because I don't understand how the checking of $2 works in other sections.

[endofzero]

Can someone test the lastest source of master, i added my changed for sync... so..

Usage : minecraft status | start [force] | stop | restart [warn] | say 'message' | tell user 'message' | logs [clean] |backup [full] | sync | cartography [sync]| biome [sync] | overviewer [sync] | update

'tell' was also added as well... but you should be able to './minecraft.sh overviewer sync' to have it update the offline folder before running...

[endofzero]

Fin~