search

roblabs / gdal2tilesp

Enhancements to tile cutter for parallelism and image format support
Other
20 stars 9 forks source link

process based parallelism. #8

Closed bolshoydi closed 7 years ago

bolshoydi commented 7 years ago

merge conflict repaired. Tested using:

On OS:

roblabs commented 7 years ago

Very fast!!

Tested with

roblabs commented 7 years ago

@bolshoydi, I created PR https://github.com/pramsey/gdal2tilesp/pull/5 so that your changes go into the original repo that I forked from.

bolshoydi commented 7 years ago

@roblabs nice to hear this!!!! the next step i would like to create gdal2mbtilesp script with multiprocessing support

roblabs commented 7 years ago

@bolshoydi, are you aware of this project for creating .mbtiles? I use it in conjunction with gdal2tiles for creating my final file.

https://github.com/mapbox/mbutil

bolshoydi commented 7 years ago

@roblabs yes, i know. Now i'm using gdal2tilesp to create tiles, then i use mbutil to create mbtiles. When input raster of small size (less 500Mb) everything fine and speed of loading images to mbtiles impressive, but when you load big amount of tiles (in my case 2.86Gb vrt of jpeg from 12 up to 21 zoom level made about 1.2millions of tiles) speed of loading (inserting) tiles into mbtiles became slow down from 2000 to 100 tiles / sec after 70,000 tiles inserted. My 8 CPUs were generating tiles for 5 hours + 5 hours of loading this tiles into mbtiles. I also tried different PRAGMA options to increase speed of tiles insertion. but nothing helps. So, i decided to combine both gdal2tilesp and mbutil scripts to write tiles directly into mbtiles.