Closed mmann1123 closed 5 months ago
I am running into an issue with large raster stacks, when I am applying my interpolator. Looks like maybe we just need to add a kw args parameter for series.apply. I will take a stab at it.
with gw.series( a_grid, transfer_lib="numpy" ) as src: # , window_size=[128, 128], src.apply( func=interpolate_nan( interp_type="linear", missing_value=missing_data, count=len(src.filenames), ), outfile=f"./{band_name}/interpolated/S2_SR_interpolated_{grid}_{dates[0].strftime('%Y_%m')}_{dates[-1].strftime('%Y_%m')}.tif", num_workers=6, # src.nchunks, bands=1, )
The directory already exists. Skipping. working on grid 0000000000-0000046592 ['./B12/S2_SR_B12_M_2023_01-0000000000-0000046592.tif', './B12/S2_SR_B12_M_2023_02-0000000000-0000046592.tif', './B12/S2_SR_B12_M_2023_03-0000000000-0000046592.tif', './B12/S2_SR_B12_M_2023_04-0000000000-0000046592.tif', './B12/S2_SR_B12_M_2023_05-0000000000-0000046592.tif', './B12/S2_SR_B12_M_2023_06-0000000000-0000046592.tif', './B12/S2_SR_B12_M_2023_07-0000000000-0000046592.tif', './B12/S2_SR_B12_M_2023_08-0000000000-0000046592.tif'] [datetime.datetime(2023, 1, 1, 0, 0), datetime.datetime(2023, 2, 1, 0, 0), datetime.datetime(2023, 3, 1, 0, 0), datetime.datetime(2023, 4, 1, 0, 0), datetime.datetime(2023, 5, 1, 0, 0), datetime.datetime(2023, 6, 1, 0, 0), datetime.datetime(2023, 7, 1, 0, 0), datetime.datetime(2023, 8, 1, 0, 0)] working on B12 0000000000-0000046592 NOTE: Dates are unknown, assuming regular time interval 86%|████████▌ | 4338/5032 [59:27<10:02, 1.15it/s] ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceeded. Use BIGTIFF=YES creation option. ERROR 1: An error occurred while writing a dirty block from GDALRasterBand::IRasterIO 86%|████████▌ | 4339/5032 [59:28<09:29, 1.22it/s] ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceeded. Use BIGTIFF=YES creation option. ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceeded. Use BIGTIFF=YES creation option. ERROR 1: [./B12/interpolated/S2_SR_interpolated_0000000000-0000046592_2023_01_2023_08.tif](https://file+.vscode-resource.vscode-cdn.net/home/mmann1123/Documents/github/YM_TZ_crop_classifier/B12/interpolated/S2_SR_interpolated_0000000000-0000046592_2023_01_2023_08.tif): WriteEncodedTile/Strip() failed. ERROR 1: TIFFAppendToStrip:Maximum TIFF file size exceeded. Use BIGTIFF=YES creation option. --------------------------------------------------------------------------- CPLE_AppDefinedError Traceback (most recent call last) File rasterio/_io.pyx:1741, in rasterio._io.DatasetWriterBase.write() File rasterio/_io.pyx:174, in rasterio._io.io_multi_band() File rasterio/_err.pyx:195, in rasterio._err.exc_wrap_int() CPLE_AppDefinedError: An error occurred while writing a dirty block from GDALRasterBand::IRasterIO During handling of the above exception, another exception occurred: RasterioIOError Traceback (most recent call last) [/home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py](https://file+.vscode-resource.vscode-cdn.net/home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py) in line 50 [126](file:///home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py?line=125) print(f"working on {band_name} {grid}") [127](file:///home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py?line=126) with gw.series( [128](file:///home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py?line=127) a_grid, transfer_lib="numpy" [129](file:///home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py?line=128) ) as src: # , window_size=[128, 128], ---> [130](file:///home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py?line=129) src.apply( [131](file:///home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py?line=130) func=interpolate_nan( [132](file:///home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py?line=131) interp_type="linear", [133](file:///home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py?line=132) missing_value=missing_data, [134](file:///home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py?line=133) count=len(src.filenames), [135](file:///home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py?line=134) ), [136](file:///home/mmann1123/Documents/github/YM_TZ_crop_classifier/2_xr_fresh_extraction.py?line=135) outfile=f"[./](https://file+.vscode-resource.vscode-cdn.net/home/mmann1123/Documents/github/YM_TZ_crop_classifier/){band_name}[/interpolated/S2_SR_interpolated_](https://file+.vscode-resource.vscode-cdn.net/interpolated/S2_SR_interpolated_){grid}_{dates[0].strftime('%Y_%m')}_{dates[-1].strftime('%Y_%m')}.tif", ... [1293](file:///home/mmann1123/miniconda3/envs/xr_fresh/lib/python3.9/site-packages/geowombat/core/api.py?line=1292) ) File rasterio/_io.pyx:1743, in rasterio._io.DatasetWriterBase.write() RasterioIOError: Read or write failed. An error occurred while writing a dirty block from GDALRasterBand::IRasterIO
I am running into an issue with large raster stacks, when I am applying my interpolator. Looks like maybe we just need to add a kw args parameter for series.apply. I will take a stab at it.