We should go back again and look at where memory is getting used, and make sure we are freeing stuff when we can.
The recent addition of background images adds at least (I believe) two more image-sized blocks of memory that get carried around. Running the top level pipeline, the background image and background noise gets created, but just kept in memory until later when all image products are written out.
With this, and all other data products, we should make sure that the variables are nulled when they can be. This will probably mean adding in the option to write out data products more often to the top level pipeline, or to write out data products from just the last operation, so that we can flush things to disk and not have to carry everything in memory.
We should go back again and look at where memory is getting used, and make sure we are freeing stuff when we can.
The recent addition of background images adds at least (I believe) two more image-sized blocks of memory that get carried around. Running the top level pipeline, the background image and background noise gets created, but just kept in memory until later when all image products are written out.
With this, and all other data products, we should make sure that the variables are nulled when they can be. This will probably mean adding in the option to write out data products more often to the top level pipeline, or to write out data products from just the last operation, so that we can flush things to disk and not have to carry everything in memory.