search

openclimatefix / metoffice_ec2

Subset Met Office MOGREPS-UK and UKV on AWS EC2
MIT License
11 stars 2 forks source link

Deal with increasing S3 cost #35

Open flowirtz opened 4 years ago

flowirtz commented 4 years ago

Now that we have a rapidly growing dataset on AWS (~100GB/week, likely more) we should look a bit more into how we can reduce our S3 bill. We are already at 3$/day - assuming that this will only grow it's better to look into this sooner rather than later.

We should think again about how frequently this data will need to be accessed and what kind of delay is acceptable to us. We should also probably think about whether we can live with loosing some of the data - if so, we should consider S3 IA One AZ.

TBD.

JackKelly commented 4 years ago

Also see issue #4

flowirtz commented 4 years ago

Ah, cool, didn't see that! Let's try to go for reducing cost without loosing data first, and then later go back to #4, I think.

flowirtz commented 4 years ago

For now we set up a lifecycle rule that moves all files to S3 Infrequent Access after 30 days. Let's keep an eye on it and see how it behaves.