Refactor model_blender to take in only necessary metadata

[stscijgbot-jp]

Issue JP-3701 was created on JIRA by Ned Molter:

The model_blender suite, which is called during calwebb_image3 by default (inside resample.resample_many_to_one), expects to take in an in-memory list of datamodels as input.  This poses a problem for the promised memory savings migration to ModelLibrary; we want to avoid loading multiple models into memory at once.  There are two options for how this could be fixed in the long-term:

• Make model_blender expect a ModelLibrary, and load models into memory as needed within that step using ModelLibrary's architecture that is designed to handle this.  Looking into the source code a little bit, it seems that the list of input models is accessed in several different places, which means that a borrow...shelve (load...save) operation would be needed several times.  This would be fairly straightforward to do, but might be slow
• Make model_blender take a list of metadata objects as input.  It appears that the data are not needed anywhere in the step, and just a list of metadata objects would obviously take up much less memory.  In that case only one borrow...shelve (load...save) operation would be needed for each model to append its model.meta to a list.  This solution takes more refactoring of model_blender but in my opinion would be the preferred way to go.

Setting this to low priority because there is a workaround (suggested by Melanie Clarke), which is to set all the data arrays of each model to be empty before passing them as an in-memory list to model_blender.  This workaround is not yet tested but seems likely to work without bad consequences in the short term.

 

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

Fixed by #8759

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

No specific testing needed from INS. This was an internal refactor.