Allow for less detailed "bottom" view

[toomuchpete]

I currently run a SMP server with a 250MB save folder, after a few hours of rendering, it produced a google map that worked, but had something like 40,000 files and took up 1.5GB of disk space. When I cut down the zoom levels, it just cropped the map, rather than just not allowing you to zoom in as far.

This had the desired effect of reducing disk usage of the map, but didn't actually show the whole map. It would be nice to have an option for this to account for really large servers.

[brownan]

Hmm, I suppose "zoom level" is a bit misleading. However, the Overviewer works by rendering the highest level tiles and then scaling them out for the other zoom levels. There's no way to directly render the lower zoom levels.

If you wanted to save on disk space and/or bandwidth, you could delete the inner most folders of tiles, but they would need to be re-generated if you ever wanted to re-render your map.

In short, sorry, but I don't think this is feasible.

[toomuchpete]

That's a shame -- it really limits the usefulness of the program for folks with even moderately-sized worlds, though I do understand the need (for caching's sake) to keep the full-size rendered chunks around.

That does raise the question though, what if the chunks were just rendered with a scale factor? Alternately, it appears that the script expects terrain.png to contain 32x32 resolution tiles -- if the script itself calculated the tile resolution, this problem could be helped along somewhat by creating a custom terrain.png with 16x16 textures or even 8x8 for really large worlds. I suspect that, without a lot of work, that would still result in a whole bunch of files, but at least they'd be smaller.

At the very least, I'd love to see a way to change the max zoom depth that gets put in the index.html to keep those files off limits to preserve bandwidth. After all: disk space is cheap, it's bandwidth that's the killer.

Thanks for your hard work -- this is a really great tool.

[brownan]

You're right, I could render the chunks scaled down. This would work, but would be a somewhat involved change since it touches so much of the code.

It's certainly a good idea though, and I'll keep this ticket open.

Just to re-iterate: the idea is to not render the furthest zoom level (or more) and instead render a higher level of tiles directly.

Actually I could do that without too much trouble now if I kept all the chunks as they are but scaled them down to render the tiles. It probably wouldn't save very much time to render, but it would undoubtedly save a lot of disk space... at the cost of not having all the zoom levels.

I am definitely going to think about this!

[gogusrl]

I've been reading about zoom levels and that it uses 7 by default so I rendered mine with 5 only to discover after a few hours that it doesn't cover everything. I tried to rerun the program with zoom level 6 but after rendering chunks I get this error

39000/39876 chunks rendered Done! Your map seemes to have expanded beyond its previous bounds. Doing some tile re-arrangements... just a sec... Traceback (most recent call last): File "gmap.py", line 144, in File "gmap.py", line 102, in main File "quadtree.pyc", line 264, in go File "quadtree.pyc", line 175, in _increase_depth NameError: global name 'imgformat' is not defined

[eminence]

gogusrl, are you using the latest version? i think that bug was fixed in 5a14f1b486892b7ba0882356faa3963151ea3a34

[gogusrl]

I'm to dumb to compile the source so I'm using the old exe. I've been trying now to fix the vcvarsall.bat error, so maybe if the stars align i'll be able to compile it myself.

[eminence]

what are you trying to compile? if you're on windows, and you have a recent version of python installed, all you should need to do is open a command prompt window and run:

c:\python26\python.exe c:\path\to\world\ output_dir

if you don't have python, you can get an easy to install version at python.org

[gogusrl]

eminence, I do have python installed, and according to readme I need PIL (Python Imaging Library) and Numpy installed. I downloaded the PIL, tried to run python setup.py install and I get a vcvarsall.bat not found error that seems to be related to microsoft visual studio stuff.

Anyway, brownan said he'll compile a new version so let's keep this clean of my noobness.

[eminence]

ah, ok. if you're curious, the vcvars bat file is used to modify your environment to get it ready to run the visual studio tools (which most importantly means that it sets the path so that cl.exe and link.exe can be found). you might try running the bat file manually if setup.py can't find it. for 32-bit machines, try: c:\program files\Microsoft Visual Studio vc/bin/vcvars32.bat (or any bat file in bin that starts with vcvars)

[brownan]

I've put out a new compiled exe, if you haven't tried that yet. (It was 5 days ago, but just in case you didn't see it)

[toomuchpete]

Revisiting this issue as my world file is now > 1.25GB in size, and the process takes a long, long time.

If I'm understanding the code I'm seeing correctly, It seems like the least obtrusive way to do this would be for the quadtree.render_worldtile to calculate its the dimensions of its image based on how many chunks it receives, and then scale the final image down to the appropriate size. Then the _apply_render_worldtiles could have some special code for the first render level and lump several of them together.

Does this sound like it would work, or would it cause major issues?

[KariTrace]

...I know I'm jumping in here mid-stream..... Could just get a bigger hard drive :)

[ghost]

I am concerned with bandwidth and disk space issues as well. It was mentioned earlier that you can just delete the closest zoomed in levels if you don't want them. How do I determine which directories to exclude? And do I have to make modifications to something Javascript-related to not make the Google Map thing panic and die?

[Silent-Bob]

i have storage issues too and have looked into it.

I would love to have a solution inside the overviewer code (preferrably by having an option to specify the min and max zoomlevels that should be generated). I tried my luck with the code, but could make my way to it. If someone knows how to do ist but lacks the time or fun doing it, please let me know how it works and i will try my luck with it.

Until then i found some ways to reduce the filesizes:

Switching to jpg instead of png drastically reduces the filesizes. If you reduce the quality via the parameter "imgquality" the filesize drops even more. Using a quality of 25 (which has notable impact on the beauty) i could reduce the disk usage per rendermode from about 1.5GB to under 200MB!!!!

To drop the closest zoomlevel i use the following perl-code:

# read zoomlevel from generated overviewerConfig.js
my $zoomlevel=0;
if (open OC,"overviewer/overviewerConfig.js")
{
  foreach (<OC>)
  {
    if (/'zoomLevels':\s+(\d+)/)
    {
      $zoomlevel=$1;
      last;
    }
  }
  print "deleting highest zoomlevel ($zoomlevel)\n";
  # create a shellglobstring like overviewer/*/*/*/*/*/*/*/*.jpg which will match the lowest level of tileimages in all rendermodes
  my $delstr = "overviewer/".'*/' x $zoomlevel."*.jpg";
  unlink glob "$delstr";
}
else
  {print "couldn't open overviewer/overviewerConfig.js\n";}

[KariTrace]

When you swapped to JPG and dropped the quality what happened to the render speed? I ask 'cause I dont lack storage but CPU power...

[Silent-Bob]

I checked the logs and i see no real difference in speed from plain png (no optimisation) to jpg (i would also think imgquality has no effect on render speed).

[XeonG]

Please add a more efficient way of rendering a map, clearly dynmap can do it without needing to render at such close detail.

[CounterPillow]

@XeonG: Okay.

Oh wait, in fact, there are people actually working on this at this very moment. It's called "oil". There's a blog post about it.

[XeonG]

So I can render the map a bit more zoomed out, get it rendered/updated faster, and have it not take up as much space or create as many files?...

I can't wait....

[XeonG]

@CounterPillow what did happen to oil? see this is still a problem, some of the solutions mentioned here did any of them get implemented? I just really don't need or even want such detailed zoom, half would be fine ie if midzoom was the base