search

aelzeiny / airflow-aws-executors

Airflow Executor for both AWS ECS & AWS Fargate
MIT License
53 stars 9 forks source link

Is this library maintained? #23

Open Braffolk opened 2 years ago

Braffolk commented 2 years ago

Hello,

This library perfectly fits out current needs, but the last commit was 8 months ago and I haven't seen activity since may. Is it still maintained and hence, safe to use or not?

Bazilbrush commented 1 year ago

yet unknown, it would be nice to see but probably the guy is busy. Maybe we should start thinking of maintaining it as a community.

aelzeiny commented 1 year ago

Hey Braffolk.

I think it's safe to say that I've stopped maintaining this library since I no longer work with Airflow on a daily schedule. It's still running in prod for hundreds of jobs at LeanTaaS, and it hasn't been maintained because nobody has complained. Up until recently I've been in denial that this is unmaintained code, but it's time to face facts. I would love to be part of any conversation of maintaining it as a community.

<rant> In my honest (and totally biased) opinion this is the superior way to be running Airflow on AWS. I say that because Celery is prone to so many issues in comparison, including large jobs nuking the cluster and the general difficulty of auto-scaling to fit one's budget and needs. The whole motivation of writing this executor was that we didn't want to continuously monitor infra and optimize for number of machines/SLAs as our pipeline continued to grow. K8 is a great executor, but man is it complicated by no fault of its own. This executor launches the exact number of machines with configurable CPU/memory when you need it, and creating an AWS Batch cluster is far easier IMO than launching a Celery/K8 cluster. But this executor is not perfect. It has its drawbacks which can be made even better by adopting orphaned tasks in AF2.0. I really want this thing to continue to live in more capable hands. </rant>