Closed vikineema closed 1 year ago
Hi @emmaai @SpacemanPaul @omad, it looks like dockerise test failed (https://github.com/opendatacube/odc-stats/actions/runs/4191291551/jobs/7265577644). Any idea why?
The connection is reset by s3. Rerun will fix it.
Hi @emmaai @Spac @omad . I've tested the latest published image from the rerun workflow and the odc-stats version is still 1.0.32. What could be causing this?
had a look, it used the cache available in ecr, since nothing has changed with the files involved in the docker image building.
Hi @emmaai, do you have an estimate of when this can be fixed? we urgently need this new image to generate S2 gm products.
I've been noticing a lot of intermittent failures in tests using external S3 data in odc-tools - especially those using s3://sentinel-s2-l2a-cogs
. I wonder if that's what's happening here too. I wonder if that bucket recently changed to a less highly-available storage class. Ideally these tests would be replaced with mocks, or locally stored data.
Maybe in the meantime, it's worth trying to set GDAL_HTTP_MAX_RETRY
somewhere?
https://github.com/rasterio/rasterio/issues/2119#issuecomment-790024225
Apparently max_retry default is 0: https://gdal.org/user/virtual_file_systems.html
Hi @emmaai, do you have an estimate of when this can be fixed? we urgently need this new image to generate S2 gm products.
Within this week.
I've been noticing a lot of intermittent failures in tests using external S3 data in odc-tools - especially those using
s3://sentinel-s2-l2a-cogs
. I wonder if that's what's happening here too. I wonder if that bucket recently changed to a less highly-available storage class. Ideally these tests would be replaced with mocks, or locally stored data.Maybe in the meantime, it's worth trying to set
GDAL_HTTP_MAX_RETRY
somewhere? rasterio/rasterio#2119 (comment)Apparently max_retry default is 0: https://gdal.org/user/virtual_file_systems.html
For the test, GDAL_HTTP_MAX_RETRY
is set here https://github.com/opendatacube/odc-stats/blob/0826115ffc577fc46d9d80b11fa1ae872c3cf61a/docker/docker-compose.yml#L21
and here https://github.com/opendatacube/odc-stats/blob/0826115ffc577fc46d9d80b11fa1ae872c3cf61a/.github/workflows/main.yml#L137
Still, the connection can be reset by s3 server, which can not be caught or retried.
Could you have a look at PR #88, it should resolve the issue.
The docker image tagged
latest
uses theodc-stats
version1.0.32
instead of the latestodc-stats
version which is1.0.33
.docker container run -it opendatacube/datacube-statistician:latest pip list| grep odc-stats
odc-stats 1.0.32