search

armadaproject / armada

A multi-cluster batch queuing system for high-throughput workloads on Kubernetes.
https://armadaproject.io
Apache License 2.0
483 stars 136 forks source link

Create an airflow instance on Kubernetes #2255

Open kannon92 opened 1 year ago

kannon92 commented 1 year ago

We need an Airflow environment that closes represents an actual Kubernetes instance.

They use a helm chart so we should also use a helm chart.

https://airflow.apache.org/docs/helm-chart/stable/index.html

a) Install the helm chart. b) Set up a dag examples repo

┆Issue is synchronized with this Jira Task by Unito

kannon92 commented 1 year ago

https://github.com/kannon92/armada-demo

I’ve created a demo repo where I run airflow locally and jobservice and point to our eks demo cluster. I’d love to get a helm chart install of airflow that works on kind.

VasuDevrani commented 1 year ago

Hey @kannon92, I'm kinda beginner in this, armadaproject I would appreciate if you could break this task down for me. Like exactly what I am supposed to do here. And, by the way I'm well versed with helm charts and kubernetes.

thanks

sync-by-unito[bot] commented 1 year ago

➤ Kevin Hannon commented:

Mostly it’s about installing airflow via a helm chart, importing the dags from that demo and have it work with Armada.

You could use the EKS demo cluster as your Armada environment.

kannon92 commented 1 year ago

There is a ticket around code coverage increase if you want more familiarity with the code base

MdSahil-oss commented 1 year ago

@kannon92 Assign this issue to me, I would like to work on it.

kannon92 commented 1 year ago

https://airflow.apache.org/docs/helm-chart/stable/quick-start.html Could be a useful guide.

Ideally I’d love a script or something that can automate installing airflow on kind and demonstrate it working with our local armada setup.

kannon92 commented 1 year ago

@ClifHouck may start looking into this.

kannon92 commented 1 year ago

https://marclamberti.com/blog/airflow-on-kubernetes-get-started-in-10-mins/ for an example on how to setup airflow on Kubernetes. It would be nice to have a way to also include our example dags and have this work for Armada.

enesonus commented 1 year ago

Hi @kannon92 I would like to work on this issue,

Do you have any suggestions about the path I should follow, what I should pay attention to before I start to work on this issue or are the links you provided sufficient? Any help would be appreciated.

sarthaksarthak9 commented 1 year ago

/assign

akagami-harsh commented 1 year ago

@kannon92 Hi, is this issue still relevant?

kannon92 commented 1 year ago

I think so but @Sharpz7 would know better.

ambuj-1211 commented 10 months ago

hi @kannon92 and @Sharpz7 I am new to the open-source world and would love to contribute to this project. What should I know about this project related to this issue as I have already done research on airflow and creating airflow instances on Kubernetes?