With the addition of ModelLibrary, the role of ModelContainer needs to be re-evaluated. ModelLibrary certainly renders the return_open and save_open options obsolete, as well as the get_sections method, since if memory usage is a concern then ModelLibrary should be used instead.
However, during discussions related to JP-3715, it has become increasingly clear that the strictness of ModelLibrary and the additional borrow/shelve code required to access models is not necessary/desired for many use-cases. For example, the calwebb_spec3 pipeline currently makes extensive use of ModelContainer but does not have the same memory issues as calwebb_image3 does. Learning to use ModelLibrary may also be a nuisance for users manipulating relatively small datasets.
Therefore, the need may remain for a lightweight, easy-to-use class for loading a list of models and association metadata. This container should satisfy the following constraints:
Loads association metadata dictionary
Is a Sequence, i.e., can be manipulated as if it were a list
Loads all datamodels into memory (i.e., use ModelLibrary instead if we care about memory usage)
Is the default data structure loaded by datamodels.open() for asn-type data
Can be used as a context manager, i.e., with datamodels.open(asn.json) as container has the expected behavior
The plan for now is to draft this container by substantially slimming down ModelContainer, and see if it satisfies the needs of the JWST pipeline.
Issue JP-3721 was created on JIRA by Ned Molter:
With the addition of ModelLibrary, the role of ModelContainer needs to be re-evaluated. ModelLibrary certainly renders the
return_openandsave_openoptions obsolete, as well as theget_sectionsmethod, since if memory usage is a concern then ModelLibrary should be used instead.However, during discussions related to JP-3715, it has become increasingly clear that the strictness of ModelLibrary and the additional borrow/shelve code required to access models is not necessary/desired for many use-cases. For example, the
calwebb_spec3pipeline currently makes extensive use of ModelContainer but does not have the same memory issues ascalwebb_image3does. Learning to use ModelLibrary may also be a nuisance for users manipulating relatively small datasets.Therefore, the need may remain for a lightweight, easy-to-use class for loading a list of models and association metadata. This container should satisfy the following constraints:
datamodels.open()for asn-type datawith datamodels.open(asn.json) as containerhas the expected behaviorThe plan for now is to draft this container by substantially slimming down ModelContainer, and see if it satisfies the needs of the JWST pipeline.