search

spacetelescope / jwst

Python library for science observations from the James Webb Space Telescope
https://jwst-pipeline.readthedocs.io/en/latest/
Other
570 stars 167 forks source link

Ramp fitting runtime in RN variance calculation #8601

Closed stscijgbot-jp closed 2 months ago

stscijgbot-jp commented 4 months ago

Issue JP-3669 was created on JIRA by David Law:

I've been using the new C-based ramp fitting code pretty extensively lately, and the one oddity that I've noticed was from processing jw02079004003_03101_00001_nis_uncal.fits

The C code runs fine, and fast (23 seconds) but then ramp fitting spends an additional 2-3 minutes before exiting from the ramp fitting routine.  I don't recall seeing a delay this lengthy before, it's usually a second or two.

Digging into a profile, it looks like the hangup is in the compute_RN_variances step (i.e., nothing to do with the C routine, but this is the next tall pole, accounting for 82% of total runtime).

Here's the time stamps:

2024-06-25 10:37:18,531 - stpipe.RampFitStep - INFO - Using READNOISE reference file: /Users/dlaw/crds_cache/jwst_ops/references/jwst/niriss/jwst_niriss_readnoise_0005.fits

2024-06-25 10:37:18,531 - stpipe.RampFitStep - INFO - Using GAIN reference file: /Users/dlaw/crds_cache/jwst_ops/references/jwst/niriss/jwst_niriss_gain_0006.fits

2024-06-25 10:37:18,572 - stpipe.RampFitStep - INFO - Using algorithm = OLS_C

2024-06-25 10:37:18,572 - stpipe.RampFitStep - INFO - Using weighting = optimal

2024-06-25 10:37:42,425 - stpipe.RampFitStep - INFO - Ramp Fitting C Time: 22.279687881469727

2024-06-25 10:40:05,451 - stpipe.RampFitStep - INFO - Results used CRDS context: jwst_1241.pmap

2024-06-25 10:40:05,453 - stpipe.RampFitStep - INFO - Results used CRDS context: jwst_1241.pmap

2024-06-25 10:40:05,553 - stpipe.RampFitStep - INFO - Saved model in jw02079004003_03101_00001_nis_0_rampfitstep.fits

2024-06-25 10:40:05,815 - stpipe.RampFitStep - INFO - Saved model in jw02079004003_03101_00001_nis_1_rampfitstep.fits

2024-06-25 10:40:05,815 - stpipe.RampFitStep - INFO - Step RampFitStep done

I'm not sure why this NIRISS example is so different that the other cases I've seen.

Testing on jw01617001001_02101_00001_nrcblong_uncal.fits (8 frames/group, 5 groups/int, 10 ints) runtime is 1 second, but the NIRISS example (4 frames/group, 20 groups/int, 6 ints) is 2.5 minutes.

stscijgbot-jp commented 2 months ago

Comment by Tyler Pauly on JIRA:

Code changes in #8697 and spacetelescope/stcal#275

stscijgbot-jp commented 2 months ago

Comment by David Law on JIRA:

Confirmed that after the merge the NIRISS example above is now completing the final stage of ramp fitting much faster (content changes evaluated in the PR).