resample step memory usage

[stscijgbot-rstdms]

Issue RCAL-732 was created on JIRA by David Davis:

When running the resample step on all 18 detectors the memory usage is quite large and can exceed 65 GB causing the program to crash.

Tests were run on dlrdmsdp01.stsci.edu which has MemTotal:       65420592 kB (65 GB) available.

The error message is

 

2023-12-05 16:00:09,974 - stpipe.HighLevelPipeline.resample - INFO - Driz parameter fillval: INDEF
2023-12-05 16:00:09,974 - stpipe.HighLevelPipeline.resample - INFO - Driz parameter weight_type: ivm
2023-12-05 16:00:09,974 - stpipe.HighLevelPipeline.resample - INFO - Output pixel scale ratio: 1.0
Traceback (most recent call last):
  File "/home/ddavis/miniconda3/envs/rcal-0.13.0/bin/strun", line 26, in <module>
    step = Step.from_cmdline(sys.argv[1:])
...
    return self.resample_many_to_one()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ddavis/miniconda3/envs/rcal-0.13.0/lib/python3.11/site-packages/romancal/resample/resample.py", line 262, in resample_many_to_one
    output_model = self.blank_output.copy()
...
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 3.21 GiB for an array with shape (2, 16246, 26517) and data type uint32
----------------------------------------------------------------------
ERROR RUNNING STEP 'HighLevelPipeline':
    Unable to allocate 3.21 GiB for an array with shape (2, 16246,
    26517) and data type uint32
----------------------------------------------------------------------

I've attached a plot of the memory usage and the log with the complete error information.

 

 

 

 

[stscijgbot-rstdms]

Comment by Eddie Schlafly on JIRA:

Some part of this is the motivation for using sky cells.  This is 18 detectors x 2 images, I presume?  Some people in RTB do want this mode but I don't think we've promised it.  Is the 18 x 2 detector coadd a requirement for B12?

[stscijgbot-rstdms]

Comment by David Davis on JIRA:

This is just 18 detectors and is from how SDF is running the pipeline for their tests.

 

[stscijgbot-rstdms]

Comment by David Davis on JIRA:

I don't think the updates to tweakwcs (0.8.6) has fixed this. See the Feb 5 memory test attachment above.

romancal                      0.13.1.dev29

tweakwcs                      0.8.6

So we need to continue to investigate.

[stscijgbot-rstdms]

Comment by Eddie Schlafly on JIRA:

The tweakwcs change is supposed to address the tweakreg memory usage issue.  The resample one is still open, I agree.