nestauk / daps_utils

Tools for setting up and running pipelines in a Data Analytics and Production System (DAPS).
0 stars 1 forks source link

[44] Daps flow & task mixins #45

Closed jaklinger closed 3 years ago

jaklinger commented 3 years ago

Replaces boilerplate in ojd_daps. The idea is to use these mixins for de-boilerplating, e.g.

from metaflow import FlowSpec
from daps_utils import DapsFlowMixin

class MyFlow(FlowSpec, DapsFlowMixin):
    [...]

For the flows all that is set is a metaflow.Parameter called production(default=False) and a corresponding property called test, which returns the boolean opposite of production, so you can write either self.test or self.production, whichever is more natural. There is also a property db_name for retrieving either "dev" or"production"` based on the test flag.

A similar setup is used for the luigi.Task mixins.