Memory usage with Detector1 pipeline

[stscijgbot-jp]

Issue JP-3610 was created on JIRA by Maria Pena-Guerrero:

Several Help Desk tickets have either directly stated or it has been found  through investigation that the Detector1 pipeline uses too much memory. 

Probably both repos the jwst and stdatamodels would have to be modified. In the stdatamodels repository, the culprit seems to be the open function, which when given a datamodel to open creates a copy or "clone" but the original is still referenced so the previous object cannot be garbage collected. Hence incrementing the number of copies and memory used in the different steps of detector1 on the jwst repo. For example, calling model2 = RampModel(model) (as is done at the beginning of detector1, and during charge_migration, etc) will create a new model ({}model2 is not model{}) however it will reference the same data ({}model2.data is model.data{}). Which links the primary memory load for both model and probably more importantly the underlying asdf objects are the same ({}model2._asdf is model._asdf{}). As the _asdf tree references effectively everything in the model their lifetimes are linked which may contribute to the growing memory load.

Here are some of the tickets in question (more may be added as they show up):

• INC0198744 https://stsci.service-now.com/nav_to.do?uri=%2Fincident.do%3Fsys_id%3D467a901f47340a10ec5b9448436d43fe%26sysparm_stack%3Dincident_list.do%3Fsysparm_query%3Dactive%3Dtrue%26sysparm_view%3DJWST%26sysparm_view_forced%3Dtrue
• INC0198484  https://stsci.service-now.com/nav_to.do?uri=%2Fincident.do%3Fsys_id%3D58e176349734829023a2fe971153afb3%26sysparm_stack%3Dincident_list.do%3Fsysparm_query%3Dactive%3Dtrue%26sysparm_view%3DJWST%26sysparm_view_forced%3Dtrue
• INC0189965 https://stsci.service-now.com/nav_to.do?uri=%2Fincident.do%3Fsys_id%3D190aab8b97b2ad5023a2fe971153afad%26sysparm_record_list%3Dassignment_group%3D8dbea402dbf2620033b55dd5ce9619ba%5Estate%3D2%5EORstate%3D3%5EORstate%3D1%5Ecompany%3D%5EORwatch_listCONTAINSa89f418ddb8f620042685434ce961990%5EORcompany%3Dc3478b5edb013e0042685434ce961951%5EORcaller_id%3Da89f418ddb8f620042685434ce961990%5EORopened_by%3Da89f418ddb8f620042685434ce961990%5EORDERBYopened_at%26sysparm_record_row%3D10%26sysparm_record_rows%3D52%26sysparm_record_target%3Dincident%26sysparm_view%3DJWST%26sysparm_view_forced%3Dtrue
• INC0198744  https://stsci.service-now.com/nav_to.do?uri=%2Fincident.do%3Fsys_id%3D467a901f47340a10ec5b9448436d43fe%26sysparm_record_list%3Dassignment_group%3D8dbea402dbf2620033b55dd5ce9619ba%5Estate%3D2%5EORstate%3D3%5EORstate%3D1%5Ecompany%3D%5EORwatch_listCONTAINSa89f418ddb8f620042685434ce961990%5EORcompany%3Dc3478b5edb013e0042685434ce961951%5EORcaller_id%3Da89f418ddb8f620042685434ce961990%5EORopened_by%3Da89f418ddb8f620042685434ce961990%5EORDERBYopened_at%26sysparm_record_row%3D38%26sysparm_record_rows%3D49%26sysparm_record_target%3Dincident%26sysparm_view%3DJWST%26sysparm_view_forced%3Dtrue
• INC0187520 https://stsci.service-now.com/nav_to.do?uri=%2Fincident.do%3Fsys_id%3De7137d954719e510ec5b9448436d4315%26sysparm_view%3DJWST%26sysparm_view_forced%3Dtrue
•  

[stscijgbot-jp]

Comment by Maria Pena-Guerrero on JIRA:

In working to speed up the emicorr for large files I ended up writing a work around that reduces the running time and memory usage for the Detector 1 pipeline. I suggest we can put it in place while we get to work on the datamodels repo for the permanent fix. Here is the jwst branch: #8588