search

openclimatefix / nwp-consumer

Microservice for consuming NWP data.
9 stars 3 forks source link

Downloading ECMWF data from s3 bucket #172

Open gabrielelibardi opened 2 months ago

gabrielelibardi commented 2 months ago

I am trying to use this repo to download data from the ECMWF NWP. I could not find any URI for the official ECMWF s3 bucket so I used this one https://registry.opendata.aws/ecmwf-forecasts/

I tried downloading with a command like this docker run -v $(pwd):/tmp/raw -e ECMWF_AWS_S3_BUCKET=ecmwf-forecasts -e ECMWF_AWS_REGION=eu-central-1 -e AWS_ACCESS_KEY_ID=PLACEHOLDER -e AWS_SECRET_ACCESS_KEY=PLACEHOLDER nwp-consumer:latest download --source=ecmwf-s3 --sink=local --from=2024-09-01 --to=2024-09-02 but the script tries to access files in ecmwf-forecasts/ecmwf which don't exist. I can work around this by changing the hardcoded ecmwf path in the code but I am not sure if this is a bug or if I am not using the repo correctly.

peterdudfield commented 1 month ago

Thanks for this @gabrielelibardi . Yea we have a paid ECMWF S3 bucket.

did you manage to solve this? You'll be getting the free ECWMF data. Perhaps there is a way to configure which folder it looks at within the bucket?

gabrielelibardi commented 1 month ago

No, but should be fairly straight forward, and cool to have.

peterdudfield commented 1 month ago

That would be amazing, thanks

peterdudfield commented 1 month ago

Please nudge @devsjc if you need any help on this