Closed ljgray closed 6 months ago
I think this is ready for a review and some thoughts on the priority implementation. I've been testing the daily pipeline config with good results, but I'm still trying to think of edge cases where this might fail/behave worse than the old implementation. Input is welcome.
The memory tools seem to be working as expected.
This is a pretty involved PR, so it may take some time to review.
Features
This PR addresses most of the issues in #181 and #152. In particular:
available
, meaning that these will only run when nothing else is able to do anythingnext
ifsetup
did not run successfullylegacy
option which will mimic the current pipeline behaviourlimit_outputs
: restrict the number of times a task can produce output in thenext
state. Once this limit is reached, the task will immediately advance its state.base_priority
: priority modifier which is added to the calculated priority. Can be negative.Two other features are added which are helpful for memory analysis:
PSUtilProfiler
which checks memory usage every 0.5 seconds in an attempt to log peak memory used by a taskRefactoring
pipeline.Manager
class related to initial task loading and pipeline creation. This does not change any behaviour, but slightly improves the distribution of responsibility between methodsRemaining:
Note: this re-implements the multi-output functionality of #263, but that has some extra tests