search

OutThere-JWST / NISPureParallel

NIRISS Pure Parallel Processing Scripts
GNU General Public License v3.0
2 stars 1 forks source link

Large memory usage when running Stage 1 Pipeline? #7

Closed TheSkyentist closed 3 months ago

TheSkyentist commented 4 months ago

For some reason there is large memory usage when running the detector 1 pipeline with lots of files. I would love to be able to solve this...

TheSkyentist commented 4 months ago

I think this is solved by c993453. Let's see.

TheSkyentist commented 4 months ago

It was not, I am going to investigate memory leakage and usage in the JWST pipeline itself.

TheSkyentist commented 4 months ago

This makes a big difference to leo-02, since these are our largest files.

TheSkyentist commented 4 months ago

Issues have been raised in the JWST pipeline.

TheSkyentist commented 4 months ago

e0b2ea1 Sacrifices some CPU speed to reduce memory usage. A little clunky but gets the job done.

TheSkyentist commented 3 months ago

Pulling from HEAD with stpipe now vastly limits overall memory usage. Potentially reverting e0b2ea1?

TheSkyentist commented 3 months ago

I think this is more or less solved. We still need limits on Stage1, but it's relatively fast so I'm no longer worried about it.