Closed TheSkyentist closed 2 months ago
I think this is solved by c993453. Let's see.
It was not, I am going to investigate memory leakage and usage in the JWST pipeline itself.
This makes a big difference to leo-02, since these are our largest files.
Issues have been raised in the JWST pipeline.
e0b2ea1 Sacrifices some CPU speed to reduce memory usage. A little clunky but gets the job done.
Pulling from HEAD with stpipe now vastly limits overall memory usage. Potentially reverting e0b2ea1?
I think this is more or less solved. We still need limits on Stage1, but it's relatively fast so I'm no longer worried about it.
For some reason there is large memory usage when running the detector 1 pipeline with lots of files. I would love to be able to solve this...