search

dshean / demcoreg

Utilities for DEM and point cloud co-registration
MIT License
114 stars 45 forks source link

Final Mask index out of range and Dirty block error #7

Closed ShashankBice closed 5 years ago

ShashankBice commented 6 years ago

Hi @dshean , A colleague in India ran into some trouble for "IndexError: boolean index did not match indexed array" and then due to dirty block error because of the apparently large source DEM (nl: 15126 ns: 9464) after 10 iterations and tiltcorr due to memory issues as (https://github.com/mapbox/rasterio/issues/307). I implemented a new static mask calculation after first round of co registration and open the source DEM dataset in update mode to overcome the above problem. Opening dataset in update mode also avoids the seemingly file not open for writing error in the end of the code. Not sure, if this is the best way, but works for now! Please review..

dshean commented 5 years ago

First of all, I would not recommend running this tool for very large rasters, especially when RAM is limited. This was mostly developed for quick alignment of low-res DEMs (10-30 m posting).

dshean commented 5 years ago

The mask dimension issue has been bothering me for a while. Using the original mask was intended to be a shortcut, but sometimes the dimensions of shifted dataset were off by a few pixels.

Thanks for taking that on. Your fix led to some issues with the output histogram (lots of 0 values), and we needed to combine the static_mask with the difference map mask. I made these changes and verified that we're no longer seeing failures due to different array dimensions.

dshean commented 5 years ago

The update issue error has also been bothering me, but it was nonfatal, so I ignored. To summarize, dem_align.py would output:

Exception ignored in: RuntimeError: TIFFWriteEncodedTile:File not open for writing

We definitely don't want to modify the original input files, so we don't want to open in update mode. Even if we're not actually modifying to the original file, your fix led to a change in file modification time.

I reviewed the dem_align code and couldn't figure out where we were attempting to modify a dataset that wasn't a temporary copy. We open dem2_ds and immediately make a mem_ds copy that we update, so we never attempt to modify the original dem2_ds. I cleaned up code to make sure we're closing datasets when no longer needed. I thought the issue was related to something with the GDAL MEM driver CreateCopy method.

In the end, I tracked it down to the hillshade generation for the final figure, which was calling geolib.gdaldem_mem_ma. That function was modifying the input dataset in place, rather than creating a copy. I changed here: https://github.com/dshean/pygeotools/commit/35949ddaeb5f17a4a3ce638665b59d1f4fc5590f and changed the call in dem_align.py as well.

ShashankBice commented 5 years ago

Great! I did not notice the fine changes caused by the edits. Will be careful next time. The code runs well with the both normal and the unusually large data sets now!