Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
Apache License 2.0
17.71k
stars
2.39k
forks
source link
Optimized _partition_tasks function by reducing iterations over task_… #3261
Refactored the _partition_tasks function in execution_summary.py to improve performance. The function now iterates over task_history only once, reducing the computational complexity and enhancing the speed of task partitioning.
…history
Description
Refactored the _partition_tasks function in execution_summary.py to improve performance. The function now iterates over task_history only once, reducing the computational complexity and enhancing the speed of task partitioning.