OverflowError: strings are too large to concat

[corvidian]

When I try to run the tool on our SMP world, I get the following error.

Traceback (most recent call last):
  File "/usr/local/bin/mian", line 9, in <module>
    load_entry_point('mian==0.8.7', 'console_scripts', 'mian')()
  File "/usr/local/lib/python2.6/dist-packages/mian-0.8.7-py2.6.egg/mian/mian.py", line 249, in main
    mian(world_dir, bt_hexes, nether)
  File "/usr/local/lib/python2.6/dist-packages/mian-0.8.7-py2.6.egg/mian/mian.py", line 176, in mian
    raw_blocks += nbtfile['Level']['Blocks'].value
OverflowError: strings are too large to concat

The world is about 94k chunks. The command line used was: mian -b 38,15,0E world

[pepijndevos]

I think this will require a rewrite to use StringIO or another kind of buffer, rather than concatenating. If you're feeling experimental, you could also try the Clojure port of this code: https://github.com/pepijndevos/Clomian

[corvidian]

Seems like the real problem was memory starvation. I ran mian on the same world on one of the workstations at our school. It ran fine for some time, but used more and more memory all the time. After a while it used ~6 gigs of virtual memory, and spent all of it's time swapping.

Is it storing all of the map data in one string or something? Wouldn't it be better to do it one chunk at a time, so the memory requirement is constant regardless of map size?

[Fenixin]

My branch count-chunk solves this problem, it makes count the chunks individually, so RAM usage keep constant. I've done a pull request.

Please, give it a try!

[l0b0]

Fixed by Fenixin@d6d01308a752e964a8f69534c1f67ccae5be12de - Thanks!