S2A + S2B products downloaded via API are significantly larger than they should be

@tyhayward thanks for this. The data, as in the raw pixel values contianed in the primary geotiff is identical between the two download methods. However, the difference you observe is that the current API can only access the data via an OGC Web Processing Service (WPS).

What happens is this:

all analysis ready data (ARD) products are stored in an Azure storage account in a Cloud Optimised GeoTIFF (COG) format.
When a user, via the portal UI, requests a download, a copy of the COG ARD (which is compressed and contains other optimisations) is provided by the download link. You receive the "smaller" file size.
However, when a user makes an API call using this WPS wrapper library or directly against the WPS end point, the WPS server, namely, geoserver cannot maintain the COG optimised format and copies the raw GeoTiff format (hence it is much bigger in terms of filesize). This is a limitation of WPS service currently.

To workaround this limitation, I've actually included some rasterio commands which add the optimisations back into the downloaded result, see eods-example3-query-eods-download-results-mosaic-s2s-with-rasterio.ipynb

You could adapt the cells (copied below):

# add overviews (pyramids) to the download this has to be "inplace" change
rio_overviews_cmd = ['rio', 'overview', '--build', '2,4,8,16', <input-raw-download-file>]
subprocess.run(rio_overviews_cmd, capture_output=True)

# convert the wps "raw" geotiff to optimised format geotiff
rio_convert_cmd = [
    'rio',
    'convert',
    <input-raw-download-file>,
    <output-cog-format-geotiff>,
    '--co',
    'compress=LZW',
    '--co',
    'tiled=true',
    '--co',
    'blockxsize=256',
    '--co',
    'blockysize=256',
]
subprocess.run(rio_convert_cmd, capture_output=True)

We have a story in our backlog to deliver the COG (optimised) format geotiff directly to the user via the API. Please feel free to complete this feedback form to register that you would like this feature, feedback form here https://defragroup.eu.qualtrics.com/jfe/form/SV_cNHH8Hp5uXOCLiJ

defra-data / EODS-API

S2A + S2B products downloaded via API are significantly larger than they should be #6