lauraduncanson / icesat2_boreal

Biomass modeling and mapping of forest biomass in the boreal using NASA's ICESat-2
14 stars 7 forks source link

Cloud Optimized Geotiff improvement #58

Open wildintellect opened 3 years ago

wildintellect commented 3 years ago

In working on Boreal-wide visualization of the dps_outputs I discovered a few things we missed while making the COGs at various steps.

  1. Not all the outputs have compression applied. Particularly the AGB outputs. The side effect is that overviews are also not compressed, so they take longer to transfer over the internet. Not a big deal until you try reading 100s of files at the same time.
  2. Not all outputs have the same number and level of overviews. Overviews are lower resolution resampling that allow dynamic mapping to operate faster when zoomed out. We should consistently create overviews to speed up map rendering. Recommended for this is 5+ (less than 128x128). Research needs to be done to figure out how to force this.

In the case of AGB outputs we can replace gdal_translate with rio cogeo create --overview-level=5 {input} {output} which will solve both the issue of compression and overviews.

Note requires rio-cogeo python library installed, conda install rio-cogeo or pip install rio-cogeo

@lauraduncanson @pahbs @nmt28 , for the F2F and COP26 I'll probably end up creating a copy of the existing data with these settings applied. For AGB it's going to shrink the files from 200mb to 82mb, and hopefully improve rendering time (3-4s per file to .5s). We should update the code for future runs to output optimized files, and add to the MAAP documentation.

wildintellect commented 2 years ago

Code will look something like

print(paste0("Write tmp tif: ", out_tif_fn))
tifoptions <- c("COMPRESS=DEFLATE", "PREDICTOR=2", "ZLEVEL=6")
writeRaster(out_stack, filename=out_tif_fn, format="GTiff", datatype="FLT4S", overwrite=TRUE, options=tifoptions)
print(paste0("Write COG tif: ", out_cog_fn))
#rio cogeo create --overview-level=5 {input} {output}
system2(command = "rio", 
        args    = c("cogeo", "create", "--overview-level=5", out_tif_fn, out_cog_fn),  
        stdout  = TRUE,
        stderr  = TRUE,
        wait    = TRUE
        )
file.remove(out_tif_fn)

to replace https://github.com/lauraduncanson/icesat2_boreal/blob/c7964d527fc6491af34b992d761a808f89964dc5/dps/alg_3-4/mapBoreal.R#L490-L494