.tif file size exceeds regular tif limits

[nkruskamp]

1: TIFFAppendToStrip:Maximum TIFF file size exceeded. Use BIGTIFF=YES creation option. (GDAL error 1)
2: C:/Users/nfkruska/AppData/Local/Temp/RtmpKO6IQ2/spat_VDJGg7EJVrX8RAq.tif: WriteEncodedTile/Strip() failed. (GDAL error 1)

I'm posting here, although this may be an issue with terra. When the data files are too large, PoPS will fail with the above warnings from R. This is likely the weather data file. It appears when the file is being copied to the TEMP location it is being written without the BIGTIFF flag set to YES which allows a .tif file to exceed 4GB. My current weather file is ~2.88GB, but it is compressed. It may be we need to ensure the .tif profile is adjusted when replicating the data (to YES or IF_SAFER), or this may fall to terra not offering that option. This is not an issue with available RAM.

Chris, this is for the 250m CA data.

[nkruskamp]

When creating a new GeoTIFF with no compression, GDAL computes in advance the size of the resulting file. If that computed file size is over 4GiB, GDAL will automatically decide to create a BigTIFF file. However, when compression is used, it is not possible in advance to known the final size of the file, so classical TIFF will be chosen. In that case, the user must explicitly require the creation of a BigTIFF with BIGTIFF=YES if the final file is anticipated to be too big for classical TIFF format. If BigTIFF creation is not explicitly asked or guessed and the resulting file is too big for classical TIFF, libtiff will fail with an error message like “TIFFAppendToStrip:Maximum TIFF file size exceeded”. from https://gdal.org/drivers/raster/gtiff.html

Underlying challenge with using compression in GTiff format. My saving data with compression leads to this issue.

[nkruskamp]

50: C:/Users/nfkruska/AppData/Local/Temp/RtmpIpJMC7/spat_4rR77HZD0RcFisK.tif, band 48: An error occurred while writing a dirty block from FlushCache (GDAL error 1)

See here for an explanation of the issue. https://gis.stackexchange.com/questions/368251/an-error-occurred-while-writing-a-dirty-block-from-gdalrasterbandirasterio

I now have my source data uncompressed with BIGTIFF set, but somewhere in our code or terra the files are being written without the correct configuration.

Admittedly, without compressing my weather coef file is now 40GB, so we may be reaching the limits of what we can do in single files?