search

openclimatefix / metoffice_ec2

Subset Met Office MOGREPS-UK and UKV on AWS EC2
MIT License
11 stars 2 forks source link

Investigate why script doesn't output files #28

Closed flowirtz closed 4 years ago

flowirtz commented 4 years ago

We get interesting file structures like the following one:

ocf-uk-metoffice-nwp/MOGREPS-UK/wind_from_direction/2020/m05/d20/h08/MOGREPS-UK__wind_from_direction__2020-05-20T08__2020-05-22T06.zarr

I'd assume that the d20/h08 stands for day 20, hour 8 - why are there files from May 22nd in there then?

Also, so far the only subfolders are for 2020/m05/d20/h08, does that mean no other files are being saved anymore? Did it crash somewhere?

Bucket Filesize as of May 29th: 122.2MB Bucket Filesize as of May 30th: 122.2MB

-> Something is wrong with the deployment - the script works locally, but deployed it doesn't output files.

Update. After some investigation it turns out that there is an IAM permissions issue. The script is not able to open files from the external MetOffice bucket.

Update 2. Added read/write permission to external MetOffice S3 via 38552aa3bc9990222a591c49ada40219931a9b77

flowirtz commented 4 years ago

Latest filesize of bucket is 123.7MB total, as of May 31st, 6pm GMT

flowirtz commented 4 years ago

It works now!!! 😍