Level 1 and Level 2 data product support in Rampviz

bmorris3 commented 3 weeks ago

This PR introduces adaptations to allow Level 2 data products to be loaded into Rampviz, at the same time as Level 1 products.

The demo video below opens L1 and L2 products for the Carina nebula. The L2 product is opened in an image viewer to the right of the usual Rampviz viewers. I've loaded the L2 product with its DQ extension, so you can see where pixels are flagged. I zoom in on a region where pixels are flagged by jump detection (pink DQ overlay), create a subset, and show that the pixels in the subset indeed have a jump.

https://github.com/user-attachments/assets/e037bfc6-f0c8-4b1f-9a0d-91976938bba9

To reproduce locally, run:

from jdaviz import Rampviz

# carina nebula:
level_2 = 'mast:JWST/product/jw02731001004_02103_00005_nrca3_cal.fits'
level_1 = level_2.replace('_cal', '_uncal')

rampviz = Rampviz()
rampviz.load_data(level_1, cache=True)
rampviz.load_data(level_2, ext=("SCI", "DQ"), cache=True)
rampviz.show()

Follow-up/to-dos

Should we load the DQ array into the group/diff viewers as well? in some cases this will be helpful, but will require careful adaptations to the data associations used for managing DQ visibility
We hope to visualize the L2 result – which is the per-pixel count rate during an integration, or the slope of the ramp cube – in the integration-viewer. L2 files are slopes from ramp fits after, e.g., linearity corrections, and without their intercepts. So we can't perfectly display this visualization from the L2 data themselves. Should we even try?
Do we want to support L2 products without L1 products? As of opening this PR, loading L2 without L1 doesn't work properly.
L1 files have no WCS, since the WCS solution is computed in the pipeline between L1 and L2. Without an L2 file, our mouseover info on ramp viewers can only show pixel coords. But if we have a matching L2 file loaded, we could show the world coordinates on hover in ramp viewers by getting the world coordinates from the level-2 viewer. Should we?
investigate why matched pan/zoom seems to "twitch" a bit in the ramp viewers

Change log entry

[ ] Is a change log needed? If yes, is it added to CHANGES.rst? If you want to avoid merge conflicts, list the proposed change log here for review and add to CHANGES.rst before merge. If no, maintainer should add a no-changelog-entry-needed label.

Checklist for package maintainer(s)

This checklist is meant to remind the package maintainer(s) who will review this pull request of some common things to look for. This list is not exhaustive.

[ ] Are two approvals required? Branch protection rule does not check for the second approval. If a second approval is not necessary, please apply the trivial label.
[ ] Do the proposed changes actually accomplish desired goals? Also manually run the affected example notebooks, if necessary.
[ ] Do the proposed changes follow the STScI Style Guides?
[ ] Are tests added/updated as required? If so, do they follow the STScI Style Guides?
[ ] Are docs added/updated as required? If so, do they follow the STScI Style Guides?
[ ] Did the CI pass? If not, are the failures related?
[ ] Is a milestone set? Set this to bugfix milestone if this is a bug fix and needs to be released ASAP; otherwise, set this to the next major release milestone. Bugfix milestone also needs an accompanying backport label.
[ ] After merge, any internal documentations need updating (e.g., JIRA, Innerspace)?

codecov[bot] commented 3 weeks ago

Codecov Report

Attention: Patch coverage is 77.08333% with 11 lines in your changes missing coverage. Please review.

Project coverage is 88.48%. Comparing base (fc88465) to head (4f21f1c). Report is 6 commits behind head on main.

Files with missing lines	Patch %	Lines
jdaviz/configs/rampviz/plugins/parsers.py	71.42%	6 Missing :warning:
...z/configs/imviz/plugins/coords_info/coords_info.py	20.00%	4 Missing :warning:
...nfigs/default/plugins/data_quality/data_quality.py	85.71%	1 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #3194 +/- ## ========================================== + Coverage 88.46% 88.48% +0.02% ========================================== Files 125 125 Lines 18677 18700 +23 ========================================== + Hits 16522 16547 +25 + Misses 2155 2153 -2 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

camipacifici commented 3 weeks ago

This looks good already as is! My preferences are those of a single user. Please gather more preferences especially from the Roman folks.

Should we load the DQ array into the group/diff viewers as well? in some cases this will be helpful, but will require careful adaptations to the data associations used for managing DQ visibility I am thinking that it might be distracting to have the DQ everywhere. I would be ok with DQ just on the L2 product.
We hope to visualize the L2 result – which is the per-pixel count rate during an integration, or the slope of the ramp cube – in the integration-viewer. L2 files are slopes from ramp fits after, e.g., linearity corrections, and without their intercepts. So we can't perfectly display this visualization from the L2 data themselves. Should we even try? The information is given in the hover, isn't it? It would be confusing to show it together with a non-corrected ramp. I would like the idea of showing the ramps without the intercept, but all the other corrections are probably out of scope.
Do we want to support L2 products without L1 products? As of opening this PR, loading L2 without L1 doesn't work properly. Imviz can do that. One day they will be combined, won't they? :)
L1 files have no WCS, since the WCS solution is computed in the pipeline between L1 and L2. Without an L2 file, our mouseover info on ramp viewers can only show pixel coords. But if we have a matching L2 file loaded, we could show the world coordinates on hover in ramp viewers by getting the world coordinates from the level-2 viewer. Should we? No strong opinion. Maybe Roman folks have preferences.
investigate why matched pan/zoom seems to "twitch" a bit in the ramp viewers Fix this is higher priority.

kecnry commented 2 weeks ago

I noticed in the screen recording that the subset layers (for a subset drawn on level 2) are not showing on the group or diff viewer - is this by design? The previews do appear when selecting them as apertures, so it seems they should be within view.

rosteen commented 2 weeks ago

You can see this in your video as well @bmorris3 but it's worse/more obvious on my machine - after doing a linked zoom (or actually, after doing anything that needs the viewers to refresh, like resizing the windows) the two default image viewers "bounce" 5-10 times. I don't see this behavior in the RampvizExample notebook on main, so I'm not sure if it's an interaction with the new viewer here, or something else.

https://github.com/user-attachments/assets/82c03408-8cba-4262-b534-469a7c8421c5

bmorris3 commented 1 week ago

@rosteen: @kecnry and I think we know where this is coming from, and we've shown that this exists on main in Imviz for viewers with different aspect ratios. The effort to fix that bug and the effort to develop the features in this PR are orthogonal, so I'm putting that bugfix in a new ticket.

rosteen commented 2 days ago

I got distracted in the middle of typing this earlier, sorry - could you add a test to check that the new viewer is created after loading level two data? Other than better test coverage I'm ready to approve this.

spacetelescope / jdaviz