industrydive / fileflow

Airflow plugin to transfer arbitrary files between operators
http://fileflow.readthedocs.io/en/latest/
Apache License 2.0
78 stars 21 forks source link

Anticipate deprecation of *args and *kwargs to BaseOperator #10

Open lauralorenz opened 7 years ago

lauralorenz commented 7 years ago

Recently running some stuff on fileflow running airflow 1.7.1.3 I was getting deprecation warnings for the following:

/Users/llorenz/Envs/fileflow/lib/python2.7/site-packages/airflow/models.py:1719: PendingDeprecationWarning: Invalid arguments were passed to DivePythonOperator. Support for passing such arguments will be dropped in Airflow 2.0. Invalid arguments were:

*args: ()

**kwargs: {'data_dependencies': {'something': 'write_a_file'}}

  category=PendingDeprecationWarning

According to the dev email list this is anticipation of only supporting explicit args:

Hi Laura,

The error is raised if an unused argument is passed to BaseOperator -- basically if there is anything in either args or kwargs. The original issue was that in a number of cases arguments were misspelled or misused by Operator subclasses and instead of raising an error, they were just passed up the inheritance chain and finally (silently) absorbed by BaseOperator, so there was no warning.

I think a workaround should be straightforward -- when you call super().init for the BaseOperator, just pass arguments explicitly rather than with args/kwargs, or (alternatively), pop arguments out of kwargs when you use them ahead of calling that init.

Our case might be a little harder because we're subclassing PythonOperator not BaseOperator directly, but we can start at this point to see how we want to workaround for Airflow 2.0. That being said, Airflow 2.0 probably isn't coming out for another... year?... so we have some time.