search

Netflix / genie

Distributed Big Data Orchestration Service
https://netflix.github.io/genie
Apache License 2.0
1.71k stars 367 forks source link

Airflow with Genie #1120

Closed Rustem closed 2 years ago

Rustem commented 2 years ago

Hi Netflix team, i am trying to integrate Genie with Airflow. So once genie is successfully deployed, we can execute different tasks like Hive/Pig. I d like to add an Airflow operator that can trigger Genie. I am wondering what would be your advice on this ? May be someone already tried to solve similar problem?

tgianos commented 2 years ago

@Rustem I've never personally used airflow so I'm not familiar with what their SDK or operator development. Looks like it's in Python? You probably want to take a look at pygenie in that case for interactions with an existing genie server.

Others appear to have attempted it in the past.

Rustem commented 2 years ago

Got it, yeah i already explored and noticed that I can easily build Airflow custom operator.