Parameterized preprocessor

etsy / boundary-layer

Builds Airflow DAGs from configuration files. Powers all DAGs on the Etsy Data Platform

Apache License 2.0

262 stars 58 forks source link

Parameterized preprocessor #33

Open y2k-shubham opened 5 years ago

y2k-shubham commented 5 years ago

The signature of process_arg(..) method of Preprocessor's hints that there's provision for passing and using arguments to preprocessor

@abc.abstractmethod
    def process_arg(self, arg, node, raw_args):
        pass

. More precisely, I want to supply some arguments to my preprocessor from within my dag-definition-YAML file .

Is that even possible?
I couldn't any example detailing how to achieve this

mchalek commented 5 years ago

This is a good question. Sadly I believe it is not possible to do this right now, but it's been a while since I have looked at this code so I'm not 100% sure. I don't have time right now but I will set a reminder to look into this again sometime soon.