search

openclimatefix / metoffice_ec2

Subset Met Office MOGREPS-UK and UKV on AWS EC2
MIT License
11 stars 2 forks source link

Do we have to use SQS? Can EC2 read directly from SNS? #8

Closed JackKelly closed 4 years ago

JackKelly commented 4 years ago

If our EC2 job stays up 24/7, then I think we could do without SQS. But I think the SNS endpoint has to be ready to receive the SNS messages as and when they are generated, otherwise they're lost.

Assuming the above is correct: I think we need SQS, which gives us the flexibility to only run the EC2 job once an hour; and also gives us resilience against the EC2 job dying for a little while.

flowirtz commented 4 years ago

Also: In case our EC2 consumer goes down, we would loose a bunch of messages. SQS is nice in that sense, as it would keep the messages for a while, once the consumer recovers it can simple go trough the old messages, without any loss.