search

etsy / boundary-layer

Builds Airflow DAGs from configuration files. Powers all DAGs on the Etsy Data Platform
Apache License 2.0
262 stars 58 forks source link

Add the python virtualenv as an operator in the boundary layer #97

Closed rfan-debug closed 3 years ago

rfan-debug commented 3 years ago

This may make python-based operator easier to use. With this operator, we will be able to define the required packages/python versions as we wish. In addition, we can also save the efforts of building a docker image and run them from grunt, especially when the task is light-weighted.

My only concern here: does the CI automatically test the correctness of this yaml? I checked the correctness by eyeballing only.

Btw, i have no idea why this operator was excluded in https://github.com/etsy/boundary-layer/pull/89

rfan-debug commented 3 years ago

According to @vchiapaikeo's suggestion, the virtualenv operator might be abused to run intensive jobs, which may probably pull down the entire airflow service. Thus we decided to close this PR.