Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
Apache License 2.0
17.71k
stars
2.39k
forks
source link
Feature request: Include completed task runs in task history #3288
Currently task history only tracks tasks that actually run - it would be useful to track when a task is scheduled, then skipped since it was already complete. The worker does communicate with the scheduler in this case, so it should be just some added logging.
Currently task history only tracks tasks that actually run - it would be useful to track when a task is scheduled, then skipped since it was already complete. The worker does communicate with the scheduler in this case, so it should be just some added logging.