search

nasa-nccs-hpda / srlite

surface reflectance
0 stars 1 forks source link

Address Maximum TIFF file size exceeded. #25

Closed gtamkin closed 1 day ago

gtamkin commented 3 months ago

If your data really is approaching the BigTiff limit, then consider splitting it up and using the VRT format. I use VRT where my rasters approach a TB let alone a PB! VRTs are brilliant and you can effectively avoid many massive operations (e.g. merges and bulk warps) by using them: https://gis.stackexchange.com/questions/244524/should-i-always-be-using-bigtiff-yes-as-default:

(ilab-tensorflow-srlite) gtamkin@ilab213:/explore/nobackup/people/gtamkin/dev/srlite/test/test-alaska-split4_v1_no_csv/20240608$ du -h /explore/nobackup/projects/ilab/data/srlite/products/srlite_1.0.1/toa/alaska_2nd_batch/split_4/WV02_20170906_M1BS_103001006F1F5D00-toa.tif 12G /explore/nobackup/projects/ilab/data/srlite/products/srlite_1.0.1/toa/alaska_2nd_batch/split_4/WV02_20170906_M1BS_103001006F1F5D00-toa.tif

(ilab-tensorflow-srlite) gtamkin@gpu001:/home/gtamkin$ /home/gtamkin/.conda/envs/ilab-tensorflow-srlite/bin/python /home/gtamkin/_SRLITE-dev/src/srlite/view/SrliteWorkflowCommandLineView.py -toa_dir /explore/nobackup/projects/ilab/data/srlite/products/srlite_1.0.1/toa/alaska_2nd_batch/split_4/WV02_20170906_M1BS_103001006F1F5D00-toa.tif -target_dir /explore/nobackup/projects/ilab/data/srlite/ccdc/ccdc_20230807_alaska_batch23/alaska -cloudmask_dir /explore/nobackup/projects/ilab/data/srlite/products/srlite_1.0.1/cloudmask/alaska_batch_2/split_4 -bandpairs [[\'blue_ccdc\',\ \'BAND-B\'],\ [\'green_ccdc\',\ \'BAND-G\'],\ [\'red_ccdc\',\ \'BAND-R\'],\ [\'nir_ccdc\',\ \'BAND-N\'],[\'blue_ccdc\',\ \'BAND-C\'],\ [\'green_ccdc\',\ \'BAND-Y\'],\ [\'red_ccdc\',\ \'BAND-RE\'],\ [\'nir_ccdc\',\ \'BAND-N2\']] -output_dir /explore/nobackup/people/gtamkin/dev/srlite/test/test-alaska-split4_v1_no_csv/20240617-gpu001-v2 --regressor rma --debug 1 --pmask --cloudmask --band8 --cloudmask_suffix toa.cloudmask.tif --target_suffix ccdc.tif --csv_dir None . . . . Apply coefficients to 8-Band High Res File... /explore/nobackup/projects/ilab/data/srlite/products/srlite_1.0.1/toa/alaska_2nd_batch/split_4/WV02_20170906_M1BS_103001006F1F5D00-toa.tif

Created COG from stack of regressed bands... /explore/nobackup/people/gtamkin/dev/srlite/test/test-alaska-split4_v1_no_csv/20240617-gpu001-v2/WV02_20170906_M1BS_103001006F1F5D00_sr_02m.tif Exception ignored in: Traceback (most recent call last): File "/home/gtamkin/_SRLITE-dev/src/srlite/view/SrliteWorkflowCommandLineView.py", line 106, in main context[Context.FN_COG] = rasterLib.createImage(context) RuntimeError: TIFFAppendToStrip:Maximum TIFF file size exceeded. Use BIGTIFF=YES creation option.

gtamkin commented 1 week ago

Added appropriate driver option to handle large files to RasterLib.py: ds_toa_copy_GTiff = driver_GTiff.Create(intermediate_output_name, xsize=ds_toa.RasterXSize, ysize=ds_toa.RasterYSize, bands=numBandPairs, eType=toa_datatype, options=['COMPRESS=LZW','BIGTIFF=YES'])

gtamkin commented 1 day ago

Fixed and tested with existing TOAs. Word has it that much larger pan-sharpened TOAs are coming which will most likely break SR-Lite. This issue can now be closed and a different issue will be created. See: https://github.com/nasa-nccs-hpda/srlite/issues/29