radiocosmology / caput

Cluster Astronomical Python Utilities
GNU General Public License v2.0
12 stars 21 forks source link

Pipeline runner improvements #258

Closed ljgray closed 6 months ago

ljgray commented 11 months ago

This is a pretty involved PR, so it may take some time to review.

Features

This PR addresses most of the issues in #181 and #152. In particular:

Two other features are added which are helpful for memory analysis:

  1. A function to find the total memory footprint of an object, including any objects it references.
  2. A monitoring function in PSUtilProfiler which checks memory usage every 0.5 seconds in an attempt to log peak memory used by a task

Refactoring

Remaining:

Note: this re-implements the multi-output functionality of #263, but that has some extra tests

ljgray commented 10 months ago

I think this is ready for a review and some thoughts on the priority implementation. I've been testing the daily pipeline config with good results, but I'm still trying to think of edge cases where this might fail/behave worse than the old implementation. Input is welcome.

The memory tools seem to be working as expected.