search

hyperbadger / nomad-pipeline

Run pipeline-style workloads in Nomad
MIT License
42 stars 2 forks source link

Integration between nomad-pipeline and apache airflow #3

Open dob3001 opened 2 years ago

dob3001 commented 2 years ago

It might be worth considering some type of integration between nomad-pipeline and airflow...

This is the k8s provider for airflow.

https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/index.html

This could potentially mean that airflow can use nomad/nomad-pipeline to create DAGs etc. I have not specifically checked the functional ask of k8s so I'm assuming it's not too different in what we can do in nomad/nomad-pipeline.

dob3001 commented 2 years ago

https://github.com/apache/airflow/tree/main/airflow/providers/hashicorp <- there is one for vault integration

jsp-hashicorp commented 1 year ago

Any update for this integration request?

waquidvp commented 1 year ago

Currently I am not sure about what the integration with Airflow would look like. I am open to suggestions and would be happy to implement.

mSALDANHAf commented 1 year ago

Currently I am not sure about what the integration with Airflow would look like. I am open to suggestions and would be happy to implement.

Mainly the hability to run jobs (usually of batch type)