Calibrate images based on single integration ramps and pair-wise differences between groups

[stscijgbot-jp]

Issue JP-2941 was created on JIRA by Alicia Canipe:

There is interest from observers in getting calibrated images based on individual integrations and on pair-wise differences between consecutive groups within integrations. In particular, the pipeline currently results in smeared images for objects or features that are not fixed on the detector during exposures (best case), or removes some or all of the signal from them during outlier rejection steps. This request is for the pipeline to produce calibrated images equivalent to i2d products from each individual integration (e.g. _i2dints.fits) and from pair-wise differences between groups within each integration (e.g. _i2dgrps.fits).  Such products would have utility for moving-target science, and may have broader applicability for time-domain science.

Such products would support 2 moving-targets science use cases:

For tracked (i.e. moving target) observations of complex systems, various features (e.g. clouds, moons) move at significantly different rates than the tracked body itself. Such products will enable high spatial resolution images of such features and retrieval of accurate flux and astrometric measurements.

For fixed-target observations, such products will enable detection, and retrieval of accurate flux and astrometric measurements, for moving targets serendipitously present in the scene.

There may be utility in providing similar data for the IFU modes of NIRSpec and MIRI, although the need for such products seems to be less urgent.

[stscijgbot-jp]

Comment by Anton Koekemoer on JIRA:

Thanks Alicia Canipe for filing this ticket. One question, to help get things started, could we have a couple more details please on why the current multi-group products before ramp fitting (ie rateints.fits, and/or ramp.fits) are not useable for this? Having that info would be helpful in deciding the next steps. Thanks!

[stscijgbot-jp]

Comment by John Stansberry on JIRA:

Hey Anton,

I'm not familiar with ramp.fits (not a standard product?). It looks like it may be the bias subtracted and linearized ramp. As such it's better for science than uncal.fits, but isn't really what people are asking me for.

Rateint.fits is as desired in its structure and temporal sampling, but lacks flat-fielding, conversion to physical units and rectification. A 'calints.fits' product would address the first 2 issues, and an 'i2dints.fits' product would address all of them.

There are some aspects of this issue that call for even higher time-resolution than can be achieved at the individual integration level. For example, many asteroids can cross a NIRCam SW pixel (for a fixed pointing) in a frame time (or less). Similarly, cloud features in Jupiter's atmosphere are smeared in even relatively short integrations. Applications like these would benefit from having calibrated products (cal, i2d) computed from 2-group differences within integrations. I've notionally called those 'calgrp.fits' and 'i2dgrp.fits' in the Description.

[stscijgbot-jp]

Comment by Bryan Holler on JIRA:

Below is feedback from Mark Showalter (SETI), a member of the solar system ERS team studying the minor moons of Jupiter, on pipeline topics related to this ticket. It would be good to have a conversation (internally) at some point to discuss feasibility and priority.

The problem, as you know, is that whenever an object is moving relative to the FOV, it gets lost in the pipeline even though it shows up clearly in the individual groups. When we process the data for Jupiter's tiny moons, we read the _uncal files and immediately convert to successive subtractions: group 2 minus group 1, group 3 minus group 2, etc. This provides the finest time resolution, which is what we need to eliminate the smear and see moving targets clearly. I am not sure how Ricardo handled the Jupiter cloud images, but he must have done something similar---although I suspect the clouds do not move and change quite as fast as the moons.
 
The images processed in this way look fantastic for us, except for the fact that the first frame contains the background/bias that we need to handle separately. But the problem is that we don't know the absolute calibration in physical units. We have derived some hacks to fit the calibrated pixels to the raw pixels and get a scale factor, but it is far from ideal.
 
What we really would like is for the pipeline to do a big chunk of this work for us, so we can use calibrated images that have the full time-resolution of the individual groups. It means that, whereas the pipeline normally does the "up-the-ramp" processing and then merges all the groups and iterations, it would instead give us calibrated images that are still broken down by group and integration.
 
I know that the pipeline can handle cases where the number of groups is one, so what we need, very roughly speaking, is for the pipeline to treat each pairwise-subtracted group in the _uncal file as if it were the only group. (It would also have to recognize that the pairwise-subtracted groups do not contain any background or bias, because that is removed in the subtraction.)

[stscijgbot-jp]

Comment by John Stansberry on JIRA:

Any idea why notifications of changes to JP tickets are devoid of any useful info? Here's an example:

P-2941 has been updated.

Please review any updates at https://jira.stsci.edu/browse/JP-2941.

For ~all other projects there is some indication in the notification of what got updated.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

John Stansberry I too have noticed that quirk regarding update notifications. I'll check with Stacy (Anastasia) Smith to see if she has ideas about how that could be changed.

[stscijgbot-jp]

Comment by David Law on JIRA:

I think it's just for changes to some metadata fields (e.g., Component); new comments always seem to trigger a more informative email?