Speed up raster import process and solve some issues

splashblot / dronedb

Location Intelligence & Data Visualization tool

http://carto.com

BSD 3-Clause "New" or "Revised" License

0 stars 0 forks source link

Speed up raster import process and solve some issues #82

Closed ivanprado closed 6 years ago

ivanprado commented 6 years ago

Using virtual rasters (VRT) for avoiding rewrite the raster several times.
Now intermediate rasters are just VRT files, so the just one tif file is written. That is not the case when downsampling is needed (change raster type) because a intermediate file is needed here.
As all transformation are done togheter into gdaltransform so compression is much better (it fails if compression is done at gdalwarp: the resultant tif size explodes).
Now always BIGTIFF format is used.
Saving tiffs internally with tiles
More memory assigned to gdalwarp + multi option is used.

ivanprado commented 6 years ago

Working code at https://github.com/splashblot/dronedb/tree/tileo-feature/speedup_gdal_import

Deploying into beta to test there.

ivanprado commented 6 years ago

That are the proposed changes: https://github.com/splashblot/dronedb/commit/86a13ec885d3099da865b2e0eb42474a894fc6ce

ivanprado commented 6 years ago

Noticed that the import got stuck with big rasters. Finally found the cause.

The source of problem was a statement timeout (it seems to be 5 minutes). But there where a second problem that was the one that make it get stuck instead of just failing. A library for running command is being used Open3. That library has a command to execute a pipe and then read the output of the pipe. But if was executed in a blocking fasion... So if pipe results is bigger than the output buffer then the pipe just stop waiting somebody to read the buffer.... but Open3 is waiting the pipe to finish to consume the buffer! so a deadlock I've changed the code to avoid piping (just writting intermediate files)... and now the Statement timeout arises.

Remaining tasks:

[x] Find the way to increase the statement_timeout but just for these import process, including the calls to psql and the direct SQL through the rails sql driver.
[x] Include new tests for the new code. See file raster2psql_spec.rb

ivanprado commented 6 years ago

Code is implemented and in process on being deployed to production: https://github.com/splashblot/dronedb/commit/860a05c5c50127e956f00aabbb2890933f4a7c7f

There is a remaining problem. Diego's tiff 2017-09-27_rgb_olivo_valdezarza_2_transparent_mosaic_group1.tif (3GB) is painfully slow. It is interesting that I was able to load a crop of that raster with weight 500MB pretty quickly. So probably gdal doing not the proper work here.

I have noticed that we have a pretty old gdal version: GDAL 1.11.0, released 2014/04/16. Maybe newer versions has solved that in a better way. As next step it would be nice if we can recompile a newer version and test with it to see if the problem disappears.

ivanprado commented 6 years ago

On production. Still the issue with Diego's Tiff but with a working workaround. We'll keep track of that on the issue https://github.com/splashblot/dronedb/issues/88. Closing that one.