nestauk / daps_utils

Tools for setting up and running pipelines in a Data Analytics and Production System (DAPS).
0 stars 1 forks source link

Investigate removing `talk_to_luigi` #119

Closed jaklinger closed 1 year ago

jaklinger commented 2 years ago

There is an easy way and a hard way to remove talk_to_luigi. The easy way is simply to integrate the decorator into DapsFlowMixin, and the hard way is to do away with it all together and simply discern the run ID from the logs. From a user's perspective they should be roughly identical, but the former encourages use of DapsFlowMixin, which is likely no bad thing since it comes loaded with other features which enforce daps best practice.