python freezes on windows

[bolshoydi]

when tiling small .tiffs (about 100Mb) script works fine. when tiling big .tiffs (more 1Gb spatial resolution 0.7 meters) after generating base tiles all python.exe processes became freeze. No messages in console, no generating of new tiles. *resolution in 0.7 meters means 18 zoom level, so i use command _gdal2tilesp.py mytiff.tif -z 12-18 outfolder - and this command freezes python. But when i use less zoom leveles e.g. 16 script works till the end, but Windows throws me an error .

• i've tried python 2.7.10 x32 and python 2.7.10 x64

[roblabs]

@bolshoydi

Some ideas on what to try

• Set the number of processes to 1

gdal2tilesp.py -h

has this option

--processes=PROCESSES
                        Number of concurrent processes (defaults to the number
                        of cores in the system)

• Try gdal2tiles.py, the original source which is maintained by gdal.org, as it is the latest code and may have more memory considerations. The features I use from gdal2tilesp.py, this code base, are XYZ and concurrent processes, but I admit it is out of date from the original.
• Try running a docker instance of GDAL, which is based on Linux and is straightforward to use. It will have gdal2tiles.py so you'll have to install gdal2tilesp.py. But at least you could decouple any issues away from Windows.

Install from https://hub.docker.com/r/geodata/gdal/

docker run -v $(pwd):/data geodata/gdal gdal2tiles.py

• Post your 1 GB tiff and I can give it a spin if I have time. I only have processed TIFFs in the 100-200 MB range, so I have not ran into this problem, but I'd be willing to run through the battery of testes I mention above on my machine.

[bolshoydi]

I dropped queue and progressbar. I don't know the reason, but after queue.put(tcount), queue.qsize() return 0, (in python console i tested queue.put() and get() methods, they worked fine) and queue.get() doesn't work, so i wasn't unable to get tcount from generate_base_tiles thats why i dropped out progress bar. And i added pooli.wait(), where _pooli=pool.apply_async(worker_basetiles, [argv, cpu]), and pool.terminate() before pool.join(), because operations of tiling were completed (0 cpu usage) but the processes of python.exe in taskmanager weren't closed.

• correction concerning queue i've made only for Generating Base Tiles. Overview Tiles i left as is.
• I don't know the behavior of such script on Linux, but these corrections helped me on Windows. *in the end of script i 've got windows runtime error R6016 - not enough space for thread data, but all the tiles were finished.
• Why you use queue.get(timeout=1) method with timeout=1 every time such call waiting for a 1 second and it makes script slow. I think queue.get() would save time.

[roblabs]

Glad to see you have it working. All the code changes you made may have been done in the original http://gdal.org.