JP-3639: Implement 1/f noise correction for ramp data

[melanieclarke]

Resolves JP-3639

Closes #8517

Add a flicker noise correction step to the detector1 pipeline.

Implements both a median-based correction, based on Chris Willott's image1overf, and an fft-based algorithm, based on Bernard Rauscher's NSClean. The nsclean step in spec2 is now aliased to the clean_flicker_noise step, which can support either rate-like input or ramp-like input.

Various scene masking options are provided for either correction option. When run from detector1, the ramp_fit step is run to generate a temporary rate image, used to generate the scene mask. For NIRSpec spectral data, if mask_science_regions is True, the assign_wcs and msaflagopen steps are additionally run to allow masking of the entire science area. For MIRI imaging data, if mask_science_regions is True, the flat_field step is run to retrieve NON_SCIENCE flags, used to block light from the metering structure.

Checklist for PR authors (skip items if you don't have permissions or they are not applicable)

• [x] added entry in CHANGES.rst within the relevant release section
• [x] updated or added relevant tests
• [x] updated relevant documentation
• [x] added relevant milestone
• [x] added relevant label(s)
• [x] ran regression tests, post a link to the Jenkins job below. How to run regression tests on a PR
• [x] All comments are resolved
• [x] Make sure the JIRA ticket is resolved properly

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 93.78531% with 33 lines in your changes missing coverage. Please review.

Project coverage is 61.74%. Comparing base (9f82b52) to head (0396186). Report is 4 commits behind head on master.

Files with missing lines	Patch %	Lines
jwst/clean_flicker_noise/clean_flicker_noise.py	96.65%	16 Missing :warning:
jwst/nsclean/nsclean_step.py	5.88%	16 Missing :warning:
jwst/pipeline/calwebb_detector1.py	50.00%	1 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #8669 +/- ## ========================================== + Coverage 60.75% 61.74% +0.99% ========================================== Files 375 377 +2 Lines 38367 38716 +349 ========================================== + Hits 23308 23904 +596 + Misses 15059 14812 -247 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[melanieclarke]

This is ready for initial reviews. There is one change outstanding for the NSClean implementation (#8745), which needs to merge first, but I'd like to get the rest of the code reviewed for software issues. I'll run regression tests after the NSClean change is done, unless it would be helpful to see them now.

[melanieclarke]

Regression tests started here: https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/1707/

[melanieclarke]

In the regression test results:

• Minor changes are expected for nsclean for NIRSpec spec2 and downstream products. I've inspected the differences locally and they are small, mostly due to some revisions in how the scene mask is created, including a new exclusion for large negative outliers.
• Changes from truth for the new regtest for MIRI clean_flicker_noise were just because of differences between my system and Jenkins: my local CRDS cache had the wrong emicorr reference file. I okified those changes.
• Other changes (NIRISS rate and MIRI badpix_selfcal) are unrelated.