search

NeoGeographyToolkit / StereoPipeline

The NASA Ames Stereo Pipeline is a suite of automated geodesy & stereogrammetry tools designed for processing planetary imagery captured from orbiting and landed robotic explorers on other planets.
Apache License 2.0
492 stars 173 forks source link

Stage 0 PREPROCESSING "computing statistics for left" Fail #355

Closed zhaomumu233 closed 1 year ago

zhaomumu233 commented 2 years ago

I want to generate DEM using a pair of larger stereo satellite images, which are 8G in size. In the Stage0--> PREPROCESSING,The command line has been stuck at "using downsample scale 66" for twelve hours and still does not move at all. I wonder if it is because the image is too large that the program cannot continue. Does ASP have a limit on the size of the processed satellite images? Thank you for your answer~ asp

oleg-alexandrov commented 2 years ago

ASP does have a problem with very large images, if they are stored rowwise or columnwise, rather than in tiles. Such images mess up with its cache. I put it on my todo list to find some solution.

For now, you can try rewriting the image in a way which will make it easier for ASP. The command:

gdal_translate -co compress=lzw -co TILED=yes -co INTERLEAVE=BAND -co BLOCKXSIZE=256 -co BLOCKYSIZE=256 input.tif output.tif

should be able to reorganize the images as ASP prefers. The gdal_translate tool is in the bin directory.

You can also try "gdalinfo input.tif" and look a the block size. Likely you will see something like 50000x2 or so. After reformatting as above, the blocks should be 256x256.

zhaomumu233 commented 2 years ago

I tried using your command to process the raw images and put them into ASP processing and it did work, the program doesn't get stuck in the "Stage 0 PREPROCESSING" step anymore Thank you very much for your patient answer!

oleg-alexandrov commented 1 year ago

This was a bug. When encountering a large image block (tile) in a Tif file ASP would fail and stall.

I put a fix. Both ASP and GDAL itself (outside of ASP) slow down when processing such data, but at least it won't fail. The good news is that this applies to the preprocessing step only, as the rest of the steps use intermediate data written by ASP itself.

I also added incremental progress bars and printing elapsed time for each such task that may take time (statistics and writing aligned images), and a warning is printed when such problematic images are detected suggesting the gdal_translate command to reformat the images (this tool is slow too, but at least could be done offline).