COG convesion succeeds with GDAL but not with rio-cogeo

[vatsa-asteria]

I've a couple of rasters of pixel dimensions like 300k x 150k. Most of the pixels have NODATA values. The data looks like below

The file size with LZW compression is 7 GB. When I use following GDAL commands to convert to COG, conversion happens in around an hour:

$ gdal_translate source.tif tmp.tif -co TILED=YES -co COMPRESS=DEFLATE -co BIGTIFF=IF_SAFER -co COPY_SRC_OVERVIEWS=YES
$ gdaladdo tmp.tif 2 4 8 16 32
$ gdal_translate tmp.tif cog.tif -co TILED=YES -co BIGTIFF=IF_SAFER -co COPY_SRC_OVERVIEWS=YES

But if I run rio cogeo create source.tif cog.tif, conversion gets stuck at -

Reading input: <file path>
  [------------------------------------]    0%  2d 19:51:43

I tried multiple times and killed the process after couple of hours.

I believe the method vrt_dst.read() of class WarpedVRT is trying to unpack the raster (which is originally in LZW compression). Uncompressed file would come around 200 - 300 GB. This could be causing the conversion to take days instead of an hour.

[vincentsarago]

@vatsa-asteria thanks for the issue can you try with

rio cogeo create source.tif cog.tif -p LZW --co BIGTIFF=IF_SAFER --no-in-memory --allow-intermediate-compression --co SPARSE_OK=TRUE

Note: if you have gdal>3.1 there is no real advantage of using rio-cogeo over GDAL for the conversion you're trying to do

[vatsa-asteria]

@vincentsarago with above command too, I still see the same message

Reading input: <file path>
  [------------------------------------]    0%  2d 20:58:42

[EDIT]: https://gdal.org/drivers/raster/cog.html mentions SPARSE_OK=TRUE/FALSE requires GDAL >= 3.2. I'm using 3.1. I'll upgrade GDAL and try.

[vincentsarago]

👍 , you're COG is definitely SPARSE so 3.2 will be a huge improvement

Not sure it will change anything for rio-cogeo

[vatsa-asteria]

Upgrading GDAL didn't help either. It still shows over 2-3 days as reading time.