Live ECMWF India - Githubissues

openclimatefix / nwp-consumer

Microservice for consuming NWP data.

9 stars 3 forks source link

Live ECMWF India #102

Closed peterdudfield closed 7 months ago

peterdudfield commented 8 months ago

Deploy live NWP for ECMWF India.

We are currently waiting on ECMWF to start sending us data. It will be put in the S3 bucket XXXX (on production). TODO need to see if this data is here yet.

The Nwp-consumer should then take this data (probably grib files) and save the data in the OCF foremat, (zarr.zip)
Becasue only one lot of data is being sent, we need to think how to share the data with the dev enviornment. I think there are some s3 sync options that we could do

devsjc commented 7 months ago

Waiting on ECMWF to get back in touch with details about the delivery.

peterdudfield commented 7 months ago

might need this - https://repost.aws/knowledge-center/datasync-transfer-cross-account-s3

peterdudfield commented 7 months ago

https://docs.aws.amazon.com/datasync/latest/userguide/tutorial_s3-s3-cross-account-transfer.html

peterdudfield commented 7 months ago

It might be easier to give access the nwp-consumer access to this bucket, using https://stackoverflow.com/questions/77637011/how-to-provide-access-to-s3-buckets-in-a-different-aws-account

peterdudfield commented 7 months ago

https://repost.aws/knowledge-center/cross-account-access-s3

peterdudfield commented 7 months ago

Another potential method would be to pass through aws credietnials so that nwp can read the new bucket. This might be simplier and less fiddler than creating extra roles e.t.c