GeoTIFF file sizes much larger on HPC

[Aariq]

The resulting geotiffs from the pipeline are much larger on the HPC and aux.xml files are written.

ls -lh on the HPC, running GDAL 3.3.2, released 2021/09/01:

total 9.1M
-rw-r--r-- 1 ericrscott theresam 3.1M Apr 18 11:48 normals_mean_1000.tiff
-rw-r--r-- 1 ericrscott theresam  176 Apr 18 11:48 normals_mean_1000.tiff.aux.xml
-rw-r--r-- 1 ericrscott theresam 3.1M Apr 18 11:48 normals_mean_2500.tiff
-rw-r--r-- 1 ericrscott theresam  176 Apr 18 11:48 normals_mean_2500.tiff.aux.xml
-rw-r--r-- 1 ericrscott theresam 3.1M Apr 18 11:48 normals_mean_50.tiff
-rw-r--r-- 1 ericrscott theresam  176 Apr 18 11:48 normals_mean_50.tiff.aux.xml

ls -lh on my macbook, running GDAL 3.8.5, released 2024/04/02:

total 1000
-rw-r--r--@ 1 ericscott  staff   158K Apr 18 11:20 normals_mean_1000.tiff
-rw-r--r--@ 1 ericscott  staff   171K Apr 18 11:20 normals_mean_2500.tiff
-rw-r--r--@ 1 ericscott  staff   167K Apr 18 11:20 normals_mean_50.tiff

I tried turning off the aux.xml files with "GDAL_PAM_ENABLED=NO" with no success. Need to investigate differences in default compression settings between versions. Might need to set some GDAL config stuff differently on the HPC or ask them to upgrade GDAL

[Aariq]

Tracked it down. Compression of cloud optimized geotiffs wasn't default until gdal 3.4. Turned this on in e3678b2d38cbec82b83fa007265b9b43949646a9 and now file sizes are the same. Still produces aux.xlm file 🤷‍♂️. I also requested GDAL be updated on the HPC.

total 524K
-rw-r--r-- 1 ericrscott theresam 158K Apr 18 13:34 normals_mean_1000.tiff
-rw-r--r-- 1 ericrscott theresam  176 Apr 18 13:34 normals_mean_1000.tiff.aux.xml
-rw-r--r-- 1 ericrscott theresam 171K Apr 18 13:34 normals_mean_2500.tiff
-rw-r--r-- 1 ericrscott theresam  176 Apr 18 13:34 normals_mean_2500.tiff.aux.xml
-rw-r--r-- 1 ericrscott theresam 168K Apr 18 13:34 normals_mean_50.tiff
-rw-r--r-- 1 ericrscott theresam  176 Apr 18 13:34 normals_mean_50.tiff.aux.xml