openclimatefix / nwp-consumer

Microservice for consuming NWP data.

9 stars 3 forks source link

load from s3 bucket #120

Closed peterdudfield closed 4 months ago

peterdudfield commented 4 months ago

Detailed Description

Update nwp-consumer for ECMWF data. This will load grib files straight from s3 bucket. The s3 bucket is only on production environment, so will did to give credientails to access that

Context

103 #104

Possible Implementation

Load the most recent data
join together and save
probably a new one here - https://github.com/openclimatefix/nwp-consumer/tree/main/src/nwp_consumer/internal/inputs, might be able to build of ecmwf

devsjc commented 4 months ago

Yes, since the aim is to read ECMWF data I'd be tempted to have a new file within the ecmwf adaptor e.g. s3.py that performs this.

devsjc commented 4 months ago

Files are now present in bucket - see #103 (Comment) for initial investigation into file contents.

devsjc commented 4 months ago

Quick fix to consumer for reading the files 6469ed2

devsjc commented 4 months ago

Also see: https://github.com/openclimatefix/ocf-infrastructure/pull/458