search

villasv / aws-airflow-stack

Turbine: the bare metals that gets you Airflow
https://victor.villas/aws-airflow-stack/
MIT License
376 stars 69 forks source link

Use pre-baked AMIs with installed packages and scripts #34

Open villasv opened 6 years ago

villasv commented 6 years ago

This would improve two aspects: initialization times and code reuse (worker, scheduler and interface can be identical images).

villasv commented 6 years ago

It is so tempting to dive into this. On the other hand, auto-scaling of Celery workers is not as shiny as in other domains like microservices. In fact, nowadays this seems the least relevant feature of this stack. Initialization times are always best when minimal, but are not really important.

The best motivator would be code reuse. It really is cumbersome to maintain three ~almost~ identical launch configurations. I believe that reusing the launch configuration is not enough of a motivator to a radical approach such as containers.

villasv commented 6 years ago

Code reuse can be achieved with this trick, so I don't intend to go on with images.

villasv commented 6 years ago

Reopening since Fargate makes adopting containers even more attractive (no EC2 left to manage). But I wonder if it's worth the wait for EKS to be GA and battle tested.

villasv commented 5 years ago

A year has passed, a decision must be taken. My opinion today is basically the same: it would be awesome.

But there are other areas I need to improve first. After many months of stalemate, I've decided to stick with EC2 until every other important aspect is polished and only then consider a RFC with pre-baked AMIs and maybe another RFC with containers.

villasv commented 4 years ago

After discussion on #166 I've decided to revive this issue. The WaitCondition shared metadata trick is no longer viable, the scripts were immense and were moved out of the tamplate to a shell script.

Yet another year has passed (since this was closed), and the areas I needed to improve are now better polished.