Open pahbs opened 2 months ago
A fix has been implemented that resulted in updates to build_stack.py
, CovariateUtils.write_cog()
, and CovariateUtils_topo.make_topo_stack_cog()
.
The fixes solve:
nodata
extent around tiles (now everything is clipped to tile extent)create_cutline
in write_cog
, which has been set behind if False:
for potential further development.Main features of the fix include:
topo
runs:
build_stack
) changed the input to mosaic_reader
so no reprojection is performed. This preserves north-up data for call to make_topo_stack_cog()
build_stack
) changed how in_bbox is determined - now uses the topo data's native projection - and uses total_bounds iftile is in a dateline_tile_list
write_cog
) will no longer attempt to clip during MemoryFile
stage - this wasnt working as expected. Now solely handled with align=True
write_cog
) now has a resampling
arg that defaults to nearest
make_topo_stack_cog
) now performs 2 writes [1] with write_cog
to reduce the extent a bit (but not entirely) and apply cubic
resampling
, then [2] reading back in and COGReader
which can do the final clip needed, then writing to a COG using cog_profiles
and rasterio
directly.topo_off
) not much has changed with how things work.The fix is being tested locally on mini and large dateline tiles, and regular tiles for land cover
, tree cover
and topo
input.
@pahbs would be great if you could link to a commit or PR that shows the changes.
We should also verify your outputs with rio cogeo validate
@pahbs would be great if you could link to a commit or PR that shows the changes. We should also verify your outputs with
rio cogeo validate
Here is the tag
associated with the most recent bug fixes to the build_stack
for topo
run: build_stack_v2024_2.
This tag
includes the bug fix to write correctly to a cog
using cog_translate
at the final stage of make_topo_stack_cog
.
Note:
DPS
algorithm 'run_build_stack_topo'
has been updated to run with build_stack_v2024_2 on v4.0.0 of MAAP
TODO:
DPS
algorithms run_build_stack
and run_build_stack_LC
that use build_stack
have not yet been updated.Known bugs:
64 Gb
of memory to run the meridian_tiles
that overlap the prime meridian. They will fail during the routine 8 Gb
runs. This happens because the transform to the out_crs (projected CRS) from in_crs (4326) in write_cog
during calculate_default_transform
results in a larger than normal total_bounds
. The key to fixing this will be to reduce the bounds (east-west) in a way the doesn't interfere with other non-meridian tiles runs.
@wildintellect
build_stack.py
is a tool to [1] build raster stacks for polygon extents and grids that we want, and [2] save them as COGs. There are a million ways to do this. This tool has worked well to incorporate s3 land cover, tree cover, and DEM datasets into our boreal biomass mapping workflow. We need to make sure they work perfectly for all, which means we need to fix how topo stacks are handled while maintaining currently functionality for non-topo datasets.issue:
we need to re-configure how topo stacks are handled in
build_stack
- because topography layers like slope and aspect should be calc'd on north-up data (also, we want bilinear or cubic resampling - easy to update).[1]
build_stack
intopo
mode first mosaics all intersecting DEM tiles for a given boreal tile usingmosaic_reader
(sure, we could usegdal.BuildVRT
here but it doesnt get at the heart of the issue). [2] it writes that mosaic (native north-up prj) to a COG usingwrite_cog
fromCovariateUtils.py
- which will be a tmp file [3] then usesCovariateUtils_topo.make_topo_stack_cog
to run the gdal.DEMProcessing steps - where recent updates have this working correctly based on latitude. [4] It writes out a final topo stack COG that must be clipped to theclip_geom
trick:
handling correctly the sequence of mosaicing, tmp cog writing (cuz, gdal...), topo calcs, then final cog writing into the correct prj AND with the correct clip geometry.
The issue is documented here:
/projects/shared-buckets/montesano/test_build_stack_topo.ipynb
, where we start off by reading in the tmp topo mosaic (step [2] above).write_cog
to make this work - but - how to implement this properly?rio_tiler.utils.create_cutline
inwrite_cog
doing nothing? https://github.com/lauraduncanson/icesat2_boreal/blob/b6e19887587c014afaf66f60f3ddfd453eb9d189/lib/CovariateUtils.py#L237make_topo_stack_cog
was already in theout_crs