Crash during map render

[Bytesunfish]

First and foremost, many thanks to the developers for putting out this product. I've had great experiences with it up until this point. Unfortunately, I've been wrestling with this issue for about a week to no avail. This issue seems very similar to issue #504. I hope that I'm not wasting your time with a repeat. The server seems to crash midway through the build. It isn't perfectly consistently at any particular point, but it is reliable. When the crash occurs I lose connection to the server (I access through ssh), Minecraft server dies, Apache dies, and every other remotely accessed process. The server is headless so I can't get a proper diagnosis of the extent of the incapacitance in it's current configuration. Below I've included as much relevant information as I can. Please let me know if anyone needs more information

System specs: Operating system : Ubuntu 11.04 (GNU/Linux 2.6.38-11-server x86_64) Processor: AMD 4600 Dual Core Memory: 4GB (4 x 1GB) Corsair XMS2 Swap Partition: 4GB Python version: Python 2.7.1+

Minecraft Server and Overviewer Specs: Map Size: A smidge under 3GB (not very rectangular. Has been explored in branches.) Starting script: python overviewer.py --settings=settings.py /home/minecraft/minecraft0/Render /var/www/minecraft/ Settings file: rendermode=['normal','lighting','night','cave','spawn','mineral'] verbose = 1 Mods: None Overviewer install method: Built from source

I'm going to try and do a render with 'top' running. I'll post the final stats after the crash.

[eminence]

A traceback, or other error message will be very helpful to us. Overviewer should always produce one upon a crash, so I suspect that you're simply having issues capturing it.

I'm a little confused about this ssh connection dying problem. Are you running something like screen or tmux on your server? Using one of those programs will let processes you start via ssh continue to run even if your ssh connection dies (forgive me if I'm telling you something you already know. I'm not 100% clear on how you've setup your system)

[Bytesunfish]

It's entirely possible that it's throwing an error that I'm not seeing. I'm using a windows machine to remotely connect to the headless server using Putty.

I am willing to start the overviewer in a screen if it would help, but once the crash occurs, the server is unresponsive to any and all inquiries. (all other programs running on the server also stop working)

As far as my personal experience with linux, I'm no longer a complete novice, but certainly not an expert and appreciate all inquiries and suggestions.

The system itself runs everything from samba to counterstrike servers to apache to the minecraft server.... it kinda does a bit of everything.

There are about a half dozen users that access this server on a regular basis.

If there are logs that might be helpful, please let me know, and I'll happily provide them.

I'm sad to say that I've been dealing with this error by waiting an appropriate amount of time (usually 12 hours) then forcing a hard reboot.

[eminence]

Have you narrowed this problem down to Overviewer? That is, what makes you think this is Overviewer's fault? I'm not suggesting it isn't, but any information that you have which suggests Overviewer is to blame will be useful.

If Overviewer is involved, that would be rather unusual. Typically the worst thing that happens is that Overviewer crashes, but the rest of the system continues to run happily. It's also possible that there is some other system catastrophe that is bring down everything (Overviewer included).

The next time you run overviewer, try redirecting the output to a log file. For example:

 ./overviewer.py [your options here] >& $HOME/overviewer.log

Avoid writing to /tmp, since /tmp is sometimes cleaned on a reboot.

What's the memory situation like? It's possible that Overviewer is using up enough memory to exhaust your system memory, causing the OOM killer to kill something important. This would be surprising to me, but perhaps possible.

How many cpus cores are you using with Overviewer? Have you tried running with -p1?

Do you see anything suspicious in your syslog? Typically /var/log/messages, or /var/log/syslog depending on your distribution.

[Bytesunfish]

The reason I suspect Overviewer is simply because it started happening after I installed it. Previously my system was stable. Also the issue only presents itself when I'm rendering the map.

I can't imagine that I'm running out of memory. I usually run with about 10% utilization.

I am not seeing anything suspicious in the logs. There aren't any entries near the time of the crashes.

I'll create that log file and run it. If I get a crash I'll update this with the relevant information.

I've never used the -p1 option. I've just been letting the default setting handle it. I'll continue running without it to induce the crash. After I've obtained the log file, I'll add that parameter and see what happens.

[brownan]

If it's a shared server, you certainly will want to run it with -p1 or -p2. By default, Overviewer will utilize every core it sees, which is okay for some servers, but probably not yours with lots of other stuff running.

You can also try running it with "nice" to run it at a lower CPU priority. Just put "nice" at the front of your command when you launch the overviewer.

These are only some thoughts and suggestions though. The symptoms you describe certainly seem like an out of memory scenario, but in that case, linux ought to kill the processes and keep on running. If you have a lot of swap space, I suppose it would try and eat that up before killing any processes, slowing the entire system to a standstill. There may be a way to limit the memory usage with ulimit or something but I don't know if that will work / do anything.

Another thing to do is to try rendering a smaller map and see if it still crashes. I know you probably don't want to do too much experimenting because it's a shared server. Any data you can get us will help us figure this out though. If you can get the error it throws, that would be great. If you can monitor the CPU and memory usage with top or htop and just tell us what's happening, that would also be great.

Also, what version of Overviewer are you running?

Edit: also: how much ram is available when the system isn't running overviewer? Overviewer in general is not that memory hungry, but if there's not much available, a standstill caused by memory-swap thrashing would match the symptoms you describe.

[Fenixin]

If the problem is memory starvation, as it seems to be, please, try to scan your Minecraft world with this program:

https://github.com/Fenixin/Minecraft-Region-Fixer

Some Minecraft worlds have problems that can make Overviewer (and the Minecraft server) eat all the CPU and RAM when it loads a problematic chunk. These program can report some of these problems.

[elsbrock]

Same here. Overviewer eats up way more than 500M of ram on a map with an approximate size of 200M and ends up crashing the MC server. Region-Fixer did not find any corrupted chunks. Any information on what the typical memory usage should look like?

[elsbrock]

It seems I was able to fix it using less worker processes. Memory usage seems much better now.

[eminence]

Newer versions of Overviewer should have much better memory usage. Please open a new issue if you're still having problems.