GeoscienceAustralia / dea-sandbox

Digital Earth Australia Sandbox config and planning
Apache License 2.0
13 stars 6 forks source link

Add minimum package versions for ESRI/GDAL nodata fix #288

Closed robbibt closed 4 months ago

robbibt commented 4 months ago

This PR pins minimum versions of odc-geo and eodatasets to ensure we have completely resolved the ESRI/GDAL nodata issue. Key changes include:

eodatasets3 >= 0.30.6  # min version required for ESRI/GDAL nodata fix
datacube[performance,s3] >= 1.8.19  # min version required for ESRI/GDAL nodata fix
odc-geo >= 0.4.8  # min version required for ESRI/GDAL nodata fix
robbibt commented 4 months ago

Hey @emmaai @SpacemanPaul, any idea what is causing the integration tests to fail here? It seems like they get skipped before they even run, and I can't see anything obvious in the logs... image

SpacemanPaul commented 4 months ago

I can't see anything in the logs to explain the failure either. Have you tried re-running?

emmaai commented 4 months ago

Is it necessary to copy that much auxiliary data? it blows up the disk, only 15G is available for a GitHub workflow if my memory is reliable. there is no room to pull docker image or compile one. Also the stdout is too much, it shouldn't be allowed unless debugging is required, 10s of thousands lines output will blow up any ephemeral storage for logs.

robbibt commented 4 months ago

Is it necessary to copy that much auxiliary data? it blows up the disk, only 15G is available for a GitHub workflow if my memory is reliable. there is no room to pull docker image or compile one. Also the stdout is too much, it shouldn't be allowed unless debugging is required, 10s of thousands lines output will blow up any ephemeral storage for logs.

Ah, I know what is going wrong - we copied some extra data into the tide model directory yesterday. I'll update the tests to only copy a small portion here

robbibt commented 4 months ago

Thanks @emmaai, that fixed it

emmaai commented 4 months ago

Borrowed the pr to install awscliv2 from conda, I meant to make the change a while ago. thanks!

emmaai commented 4 months ago

Thanks @emmaai, that fixed it

not sure if it did, the indentation of yaml was not right and hence the action failed. fixed in the commits

emmaai commented 4 months ago

is there any reason that geopandas<1.0? It'll cause compatible issue with Athena query of aws.