search

spacetelescope / jwst

Python library for science observations from the James Webb Space Telescope
https://jwst-pipeline.readthedocs.io/en/latest/
Other
545 stars 158 forks source link

Running `outlier_detection` with `in_memory=False` reads all models into memory #8478

Open stscijgbot-jp opened 1 month ago

stscijgbot-jp commented 1 month ago

Issue JP-3619 was created on JIRA by Brett Graham:

Running outlier detection with the recent main commit: 13e092743d6ccf9ce5352ea4d8899b726532aa4a

With an association containing 33 members (with no custom grouping or tweakreg_catalog entries for any member).

And with "in_memory=False" results in the following memory usage:

The attached graph shows that even with "in_memory=False" all models are read into memory (the largest plateau in the graph). The main cause is that the implementation of ModelContainer.models_grouped (which is used in resample) opens all models and keeps them in memory.

jdavies-st commented 1 month ago

I will add here that when in_memory=True for outlier_detection, it still writes out the blot files and then reads them back in.

2024-05-15 15:48:21,169 - stpipe.Image3Pipeline.outlier_detection - INFO - Blotting (2048, 2048) <-- (6321, 9320)
2024-05-15 15:48:21,455 - stpipe.Image3Pipeline.outlier_detection - INFO - Saved model in jw02321001009_02101_00006_nrcblong_c1001_blot.fits

So it is writing to disk when it should be doing everything in memory. This is with release 1.14.0.