Turn off the "photom" step for TSO Stage 2

[stscijgbot-jp]

Issue JP-2082 was created on JIRA by Thomas Beatty:

This mostly is a copy-and-paste from an email discussion that's been happening among some of the TSO folks on the NIRCam / STScI / ERS teams, regarding the utility of having the photom step on by default in calwebb_spec2 for TSO observations.

To quote:

I've been talking with all of you on the TSO hangouts and on some of the ERS calls about this, and there's a pretty solid consensus that this step should be turned off -- both among the NIRCam TSO'ers and the STSCI and ERS teams. For folks who haven't been there for those discussions (or aren't diehard TSO people), this is because a standard TSO analysis normalizes everything -- so you lose the information about the physical units that photom is providing anyways. Therefore in the best-case scenario where photom works perfectly, we don't gain any useful information for a TSO analysis -- practically speaking you get the same final output as if the step were turned off.

The general worry is that in actuality, because photom alters the shape of the observed spectrum, it will likely only serve to add noise to a TSO analysis. To show you what I mean, the attached plot "PhotomEffect.png" shows the multiplication factor used by photom as a function of x-pixel (I cut this down to just the region with the spectrum, since at the edges of the detector this blows up). In the pipeline the exact location of this multiplication curve is determined by the WCS solution, since really this is a function of wavelength.

One immediate problem comes from this due to pointing jitter. The expectation is that we'll get a nominal pointing stability with a standard deviation of 0.04 NIRCam pixels (~2 or 3 mas). But, those small shifts caused by pointing error aren't reflected in the WCS solution within an integration, and since the photom correction is locked to the WCS, this means that as the pointing jitters back and forth our spectra end up moving "underneath" the fixed photom correction. I ran a quick simulation to see what this would do, and it will add about 20ppm of random noise to all NIRCam TSO observations ("PhotomNoise.png").

So, in the best case photom adds nothing for TSO, but the likely case is that it will end up adding at >20ppm of noise to NIRCam (and likely the same for other instruments, though I haven't tried this). We (NIRCam instrument, ERS, STScI) therefore think that photom should be turned off for the default TSO pipeline.

[hbushouse]

I wasn't at the meeting either (due to a conflict with vacation!), but it is the case that for spectral modes the photom step uses the WCS of the data to compute the assumed wavelength of each pixel and then applies a wavelength-dependent flux calibration value. So I can see that if the data are actually shifted slightly along the dispersion direction (due to jitter), which is not accounted for in the WCS, then the assumed wavelength per pixel will also be slightly off, and hence a slightly incorrect flux cal value will get applied. Although if the flux cal varies slowly with wavelength, it might be an insignificant effect.

[stscijgbot-jp]

Comment by Sarah Kendrew on JIRA:

This change makes sense from a science point of view. I did want to add that it is important that we maintain the ability to run the photom step on TSO data for commissioning and calibration purposes. If any changes are made to this step they should still (have the ability to) be applied to TSO data, and also tested by pipeline testers. Photometric calibration of (for example) MIRI LRS in slitless mode is still part of our commissioning & calibration plans as this is an important aspect of our overall instrument characterisation. 

[stscijgbot-jp]

Comment by Jonathan Eisenhamer on JIRA:

Just a point to implementation: If done, this would most likely be handled by a parameter reference for the calwebb_spec2 pipeline and not involve any code changes.

[stscijgbot-jp]

Comment by Sarah Kendrew on JIRA:

Understood Jonathan Eisenhamer! Just wanted to have it on record that turning this step off by default does not mean we no longer care about the photometric calibration of these modes. (speaking for MIRI at least)

[stscijgbot-jp]

Comment by Karl Gordon on JIRA:

I feel that this suggestion is peripheral to the actual issue of how to update the wavelength (WCS) solution to account for the jitter. At this point, no update is done and hence it doesn't matter if the photom step is applies or not - the same relative spectroscopy results. It would be useful to learn more about how correcting for the jitter would be done. For example, would this be to update the WCS information at the beginning of SPEC2? If so, would it not be important to then include the photom step for TSO to correct for the changing responsivity as a function wavelength/pixel due to the jitter? A clear use case of how the jitter will be handled would be useful before making a decision to change how the pipeline is run by default. This may be a case where waiting for real data will really clarify the issue, including the amplitude of the impact of jitter in comparison to other detector effects.

Of course, the photom step can be turned off by the user for any offline data reduction so at least this is not a critical decision to getting science in cycle 1.

[stscijgbot-jp]

Comment by Thomas Beatty on JIRA:

Karl Gordon, yeah, I brought up the pointing jitter as one specific example of how the inclusion of the photom step could cause problems, but there's a more general concern from the TSO'ers that I talked to that photom has the potential to cause subtle errors in TSO data once we get on-orbit for no scientific or performance return. That thought process comes from the fact that the spectrophotometric reduction process for TSO data is a bit weird: we generally don't care about the shape of the spectrum out the other side, just relative changes in the measured counts as a function of wavelength. So the calibration provided by photom doesn't end up getting used, and actually, it'll get removed at the start of a TSO fitting process, as one normalizes the observations so that the out-of-transit value is unity (hence why transit depths are always in percent or ppm, but never MJy).

The general concern here is that because the photom step is directed towards getting the shape of the spectrum correct (i.e., getting the right absolute units as a function of wavelength), that transformation will act to compromise the measurement of relative flux, as a function of wavelength. Since a standard TSO fitting/analysis will remove the information provided by photom as the first step once you get your output from the "white_light" step in TSO3, that's why the thought is that including photom in Stage 2 for TSO provides no scientific benefit, but might cause problems.

I do agree that it would be interesting to think about how we could solve the pointing jitter effect as an aside, since it would be helpful to not have to back this out after the fact like one currently has to do with HST data! Likely it would require per-integration updates of the WCS at the level of <0.5 mas to bring the noise caused by jitter+photom down to manageable levels. But I'm not familiar enough with the spacecraft's operations to know if that's possible.

[stscijgbot-jp]

Comment by Karl Gordon on JIRA:

Thanks for the additional information.

I'm a bit surprised to hear that the calibration of TSO spectra - both full and relative in physical units is not something of interest. In the original discussions on the TSO data reduction with the TSO WG a few years ago, having physically calibrated spectra was requested - as well as data that would produce high quality relative measurements. This informed the development of the pipeline and - for example - is why the photom step is applied for TSO data in the baseline pipeline. As applying this step does not change the precision achieved for relative measurements this was seen as a good solution.

While I agree that this step could introduce issues in the future, it might not - especially if we continue to ensure that the guiding goal of supporting both precision relative and accurate absolute TSO use cases. So, I would advocate keeping the photom step turned on for TSO data at this point. We can always update this decision as we enhance the pipeline.

[stscijgbot-jp]

Comment by Jeroen Bouwman on JIRA:

As far as I understood this , the pipeline step applying the photom calibration only attaches a number to the data and not changing the numbers itself. Only at the spectral extraction step these numbers are taken into account to produce a flux calibrated value. or do I have this wrong? Anyway, I agree with Karl that applying a photom step should not alter the relative transit depth per spectral channel. An advantage of still having this step applied could be that one can than quickly calculate the time averaged stellar spectrum and check the flux calibration and wavelength solution.

The bigger problem I see is that the WCS, or particularly the wavelength solution is not corrected for pointing jitter. This means that for a given assigned wavelength, for each time step the true wavelength is slightly different. Especially in unresolved stellar lines or on the edges of the band passes were there are large gradients in wavelength,  this will introduce variations in flux irrespective if the photom is applied or not. For the user to be able to use the extracted spectra, a time dependent wavelength correction needs to be applied, followed by a rebin to a uniform wavelength grid if the spectral extraction step extracts per pixel row and not uses a fixed wavelength grid.

In case of intra-pixel sensitivity issues, it becomes even more difficult if we ever would want to provide a correction for this as a position dependent flatfield correction of photom value would also be required. This probably can only be derived from these datasets themself.

[stscijgbot-jp]

Comment by Sarah Kendrew on JIRA:

thanks Karl Gordon and Jeroen Bouwman. Jeroen - no, since the photom step and calibration files were updated in 2019 (... I think?), the photometric correction is now applied in the photom step itself, converting from DN/s to MJy/sr units. So the photom step output, the _cal.fits file, is a 2D image in MJy/sr. For TSOs the output is the _calints.fits files which has one such 2D image per integration. Then the extract1D step uses the pixel area information to produce a spectrum in Jy (and also applies the aperture correction factor from our APCORR reference file). 

[stscijgbot-jp]

Comment by Karl Gordon on JIRA:

The photom step does apply the calibration to the images and does not just attach this information. This was a change from the original plan to avoid multiple latter steps in the pipeline having to know how to apply the calibration.

[stscijgbot-jp]

Comment by Everett Schlawin on JIRA:

I did a quick simulation of what happens with subpixel jitter and the photom step. The issue is that the stellar spectrum moves with the jitter but the photom step's correction does not, thus it introduces a time-dependent pattern on the final spectrum. It is fairly easy to correct jitter in the dispersion direction with cross-correlation and interpolation for uncalibrated data (DN/s) but less so for calibrated dat (MJy/sr).

!photom_step_shifting_spec.png|thumbnail![^sim_photom_step_jitter.pdf]

[stscijgbot-jp]

Comment by Karl Gordon on JIRA:

Hi Everett, Thanks for investigating this issue in more detail. I am having a problem understanding why it would matter if the photom step is applied or not for correcting the jitter. The photom step is just multiplying by a pixel dependent constant as a function of time for each pixel. So it would seem to me that it shouldn't matter at all. Can you say why having a spectrum in physical units instead of instrumental units causes a difference?

[stscijgbot-jp]

Comment by Jeroen Bouwman on JIRA:

I think the problem is that applying the photom step is changing the distribution of the dispersed signal  over the detector array, especially at the long and short wavelength ends of the dispersion profile. Looking at he the posted figure sim_photom_step_jitter.pdf, a not photom calibrated extracted  spectrum, as Everett points out, looks more or less like the  bandpass plotted in the lower panel, so it will have clear edges. This has the big advantage, as the bandpass is not changing, that using a cross correlation technique with the expected spectrum for the target will give you a very accurate estimate of the wavelength shift. If you multiply with the photom file, you loose these edges and artificially increase the relative noise at the low throughput ends. Cross correlation methods will not properly work anymore in that case. I attached an real example of such cross correlation calculation for a HST/WFC3 observation, were I compare the extracted time averaged spectrum against a simple (PCE* stellar model) model for the expected signal. In this case there is no shift. To get this properly working one needs the short and long wavelength edge of the spectral profile.

!WASP-19b_icab01_check_wavelength_solution.png|width=320,height=224!

 

[stscijgbot-jp]

Comment by Anton Koekemoer on JIRA:

This was discussed in JWST Cal WG Meeting 2021-06-01, thanks everyone who participated there. Consensus was that since the pipeline currently doesn't include any WCS corrections for pointing jitter offsets, then the photometric flux calibration does not impact the accuracy of the relative photometry, so the photometry step will continue to be applied in the pipeline.

However there was also consensus that as a future enhancement, it would be good to consider incorporating into the pipeline some means of correcting the spectral dimension WCS for pointing jitter offsets (eg, cross-correlation) – and if this were to be implemented, then the topic of this ticket becomes very relevant (eg, not applying the photometric flux calibration until after the pointing jitter offsets have been corrected).

It was decided that, for now, this potential future enhancement can be passed to the TSO WG for further consideration - Nikolay Nikolov will take this back to Nestor Espinoza and Sarah Kendrew for them to add it to their TSO WG topics, being sure to include Everett Schlawin, Thomas Beatty, Jeroen Bouwman, also including NIRSpec representatives (who will be alerted to this by James Muzerolle), and being sure to also include others who may express interest.

Once the TSO WG has considered this and come up with a consensus recommendation on an algorithm for determining and correcting pointing jitter shifts that works for all the impacted instruments (including results on test data for all the impacted instruments), they can let the Cal WG know and we can schedule this for a future discussion.

[stscijgbot-jp]

Comment by Nestor Espinoza on JIRA:

I sadly had a conflicting meeting today at the same time as the CalWebb WG meeting, but to be completely honest I don't understand why the consensus was reached that the photom step has no impact on the TSO precision in the CalWebb WG meeting. Everett Schlawin's exercise above clearly shows it does. And while it is true that a flux-calibrated spectrum is something we would like to obtain for TSOs (e.g., for stellar characterization), it doesn't add anything (other than systematic noise) to relative measurements such as transit, eclipse or phase-curve observations. I'm also not sure why this was decided the way it is previously (i.e., not skipping the photom step by default). I'll try to contact the members of the TSO WG that made that decision in order to hear their thoughts/rationale.

In any case, we'll discuss this with the TSO WG in tomorrow's biweekly meeting.

[stscijgbot-jp]

Comment by Anton Koekemoer on JIRA:

Nestor Espinoza  the detailed reasoning is explained in the minutes for JWST Cal WG Meeting 2021-06-01, which also has the recording for the meeting – essentially it's because there is no correction done in the pipeline for offset pointing jitter, and Karl pointed out that the reason flux calibration was included in the pipeline was that the TSO WG had requested this in previous years.

Everett also concurred with this – his exercise above assumes that jitter pointing offset corrections are carried out (while currently they are not). If jitter pointing offset corrections are carried out, then all of us at the meeting agreed that the photometry flux calibration step should not be before such corrections, in order to avoid introducing the above issues.

So that was the idea - for the TSO WG to discuss how to include this potential enhancement of correcting for jitter pointing offsets in the pipeline in future, and as part of that discussion you would then also discuss the photom flux correction.

There are also additional complexities for NIRSpec which will I hope you will be able to include in your TSO WG discussions.

Thanks for scheduling this for further more detailed consideration in the TSO WG and also let us know please if you have further questions after reviewing the meeting notes and recording.

[stscijgbot-jp]

Comment by Nestor Espinoza on JIRA:

Thanks for the summary Anton Koekemoer! I did read the notes, but found it not as clear as you outlined here (I will watch the recording!). Will come back to this after we discuss it with the TSO WG in future meetings then.

[stscijgbot-jp]

Comment by Anton Koekemoer on JIRA:

Thanks Nestor Espinoza , I'm glad my summary here helped (I basically distilled it from the "consensus" section in the meeting notes but I also agree the meeting notes contain a lot of other material that's maybe more dense).

Thanks also Howard Bushouse for confirming that the WCS is used for the wavelength cal, and is assumed not to change. That's also the basis for the statements (above and in the meeting) that whether or not flux cal is performed won't currently impact the relative normalization results, since it's the same multiplicative factor that's being applied (as the wavelength WCS is assumed not to change) so that just cancels out.

So that's what then led to the recommendation above (and in the meeting notes) that the next step is for the TSO WG to discuss ways of measuring and correcting these wavelength shifts in the pipeline (which would also fold in much of the above discussion about the interaction with the flux photom calibration).

 

 

[stscijgbot-jp]

Comment by Nestor Espinoza on JIRA:

Hi Anton Koekemoer; we had an interesting TSO WG discussion today, which you can find here: https://outerspace.stsci.edu/display/JTEWG/2021-06-02+TSO+WG+Meeting+notes.

Overall, my take is that there was perhaps a bit of miscommunication of the effect: right now, with the photom step "on", there is an extra 20-30 ppm scatter in the simulations. We couldn't think of a good reason why it should stay on, for now, to be honest. Most measurements will be relative, and we expect that to be the norm. While we think that creating and testing a correction algorithm for the pipeline is the way to go, we believe this will most likely not get done this Fiscal Year (we have other higher priorities right now at the TSO WG). So, right now, I see two ways forward:

We leave the photom step "on" by default. Users get their lightcurves with an extra 20-30 ppm jitter; the spectrophotometric calibration is in itself also noisy, because the photom step doesn't account for jitter. This will all get fixed once we craft/test an algorithm to correct this (probably in FY2022-2023; so, a while from now).

We turn the photom step "off" by default. Users get their final lightcurve with precisions defined by other effects (and not injected by the photom step); they can turn on (or do their own) spectrophotometric calibration in the meantime. Once we craft and test an algorithm, turning the step "on" would give the correct, final spectrophotometrically calibrated spectra.

Despite the consensus reached at the CalWebb WG, I think we all agreed that we would like to push for 2. I saw Karl Gordon's concern was to keep what was decided by previous members of the TSO WG. Would you folks feel better if I reached out to them to ask what their rationale was for this? I can then touch base with you after I discuss this with them.

Let me know what you folks think!

[stscijgbot-jp]

Comment by Karl Gordon on JIRA:

Nestor Espinoza Thanks for the details from the TSO WG meeting. It would be good to reach out to previous TSO WG members to understand why the previous recommendation is different from what is recommended now. Nikole Lewis is the person I had the most discussion in the past on this topic. Not required, just would be nice to make all the use cases are considered.

I still feel I am missing something in the summary given above. Maybe we could chat on the phone/video in the next day or so briefly to more effectively close the loop (I'll message you on slack to see if we can find a time). I wonder if I'm missing the point that the correction for the jitter scatter that is done outside of the pipeline is easier/more accurate w/o the photom step.

[stscijgbot-jp]

Comment by Anton Koekemoer on JIRA:

thanks Nestor Espinoza for the update, and I agree with Karl Gordon that it would be good to make sure we have in hand the reasons for why the flux correction was requested (with all instruments represented).

Now, in terms of whether this extra "noise" is actually introduced by the photom calib (as opposed to the jitter), could I ask Everett Schlawin please to run two versions of the simulation, with everything the same (ie the same jitter, and the same processing, ending with a ppm measurement done in the same way, on the final normalized spectra), except:

(1) with the photom step not applied (ie the new proposed change)

(2) with the photom step applied (assuming a constant WCS as currently in the pipeline, ie applying the same photom correction to all the images)

In particular it would be helpful to see whether the resulting ppm in the normalized spectra is the same for these 2 cases (which sounds like it should be the case, since the same photom flux cal is applied so it should just cancel out), and that would be helpful for informing the discussion going forward.

 

 

[stscijgbot-jp]

Comment by Thomas Beatty on JIRA:

Hi all,

I wanted to clarify a bit about the concerns here, since as Nestor Espinoza and Karl Gordon said, I think we haven't communicated this well.

Here's an example I just ran that illustrates the problems with photom as currently implemented. I'll punch out the steps individually of what I did:

Take a representative spectrum from the simulations we've run, shift it by some small amount in the x-pixel direction

Apply the photom correction to the shifted spectrum (note the WCS that photom uses here is constant, as per Howard Bushouse, and does not shift along with the spectrum),

Sum the same set of pixels (i.e., the spectral extraction region does not shift along with the shift in spectrum),

See how the measured intensity changes as the spectrum shifts in pixel-space underneath the photom correction (which is constant in pixel-space).

What you get out the other side is a linear change in intensity for these small x-pixel shifts, at a rate of about +/-20 ppm for a shift of +/-0.05 x-pixels.

Our primary concern is about these apparent changes in measured intensity that occur when the spectra have sub-pixel shifts on the detector and the photom step is applied. I think we all agree that if everything is perfect, and the spectra stay in exactly the same place in each integration, then the photom step cancels out and causes no effect. But we expect to see shifts with a standard deviation of +/-0.05 x-pixels, per integration, from pointing errors. If the spectrum moves around at all in the x-pixel direction from one integration to the next, the photom step does not cancel out, and you get these spurious intensity changes using the current Stage 2 TSO pipeline. (bolded that just in case some folks are scanning these comments)

Reading over the Jira comments from the last few days, I think the disconnect here may be that this came across as some sort of issue with jitter correction that's somehow effected by converting DN/s to MJy, rather what it actually is: which is about added noise in the output TSO lightcurves caused by running the photom step.

!PhotomTest.png!

[stscijgbot-jp]

Comment by Karl Gordon on JIRA:

Thomas Beatty My disconnect is that if you don't apply the photom step, does the spectra not vary at the same level? Just now the variation is in DN/sec/pixel instead of MJy/sr?

[stscijgbot-jp]

Comment by Karl Gordon on JIRA:

Thomas Beatty Everett and I have scheduled a chat on this topic for Tuesday afternoon at 4 pm Eastern. Maybe you could join us? I feel like I am missing something critical about the work you both are doing. And very much would like to find out what it is! I feel that comments on this issue aren't working and feel talking "in person" would be better. If you are busy Tuesday afternoon, hopefully we can setup a separate time next week to chat. Thanks.

[stscijgbot-jp]

Comment by Thomas Beatty on JIRA:

Karl Gordon, that sounds like a good idea: more efficient than going back and forth here.

4pm ET next Tuesday works for me!

[stscijgbot-jp]

Comment by Anton Koekemoer on JIRA:

I also agree it sounds like a live chat between a few people would be helpful.

As input for that, might Thomas Beatty be able to run another version of the above, but just skipping step 2 completely? Ie still do steps 1, 3, 4 and produce the same plot (ppm as a function of x offset from -0.15 to +0.15 pix) but without having applied any photom correction at all?

[stscijgbot-jp]

Comment by Thomas Beatty on JIRA:

Anton Koekemoer, yeah, that's a good demonstration of what's going on here. I ran the above with and without photom, and trying to track the offsets and not tracking the offsets (plots attached).

When you don't track the offsets, then as expected both with and without photom you get intensity changes (since we're extracting slightly different regions of the spectrum each time).

When you do track the offsets, the without photom case shows zero intensity change (also as one would expect), but the with photom case ends up showing more variation. That's because as you try and track the offsets in the extraction, while you are following the same section of the raw spectrum, you're also extracting on slightly different regions in the photom correction. Since the photom correction isn't perfectly flat, shifting around under the correction gives slightly different "integrals" (though really it's the integral of spectrum*photom) at different positions, which causes the apparent intensity variation.

!Photom_DontTrackOffsets.png|thumbnail!!Photom_TrackOffsets.png|thumbnail!

[stscijgbot-jp]

Comment by Anton Koekemoer on JIRA:

thanks Thomas Beatty those plots help resolve what I think the disconnect was (at least from a perspective outside the TSO WG) – namely this extra step of 'tracking the offsets'.

It looks like we can actually agree that if the offsets aren't corrected (or 'tracked') the resulting ppm change is about the same, with or without the photom correction, ie your left-hand plot – and this is what Karl Gordon and myself were also describing when we say the photom correction should effectively divide out if the offsets aren't accounted for, ie the ppm values should be about the same.

Then when you do your extra step of correcting (or 'tracking') the offsets, first doing the photom step leads to the ppm scatter shown in the red line of your right-hand plot, and I'd say we can agree on this as well.

So the 'disconnect' (from a perspective outside the TSO WG) was that this step of correcting / tracking the offsets is an additional step, not in the pipeline (currently at least), and is part of your post-processing – so if the pipeline products are flux calibrated then this doesn't give as good results with your offset tracking/ correction as you'd get from running the pipeline offline, without the photom correction (ie your right-hand plot). Thomas Beatty would you say this captures it sufficiently?

[stscijgbot-jp]

Comment by Sarah Kendrew on JIRA:

Quick side question, Thomas Beatty - is your "x-pixel direction" along the dispersion axis, or cross-dispersion? 

 

And can you pls send me the details for your call about this? I'd like to listen in and I can make Tues @ 4 pm. Thanks!

[stscijgbot-jp]

Comment by Thomas Beatty on JIRA:

Anton Koekemoer, yes, I agree with your summary. Glad we're converging on what's going on!

Sarah Kendrew, yes, for NIRCam TSO the x-pixel direction is the dispersion axis. And I'll forward you the call invite from Karl right now (and please ignore that I apparently forget how to spell "Sarah" in that email...).

 

 

 

[stscijgbot-jp]

Comment by Anton Koekemoer on JIRA:

Thomas Beatty thanks for confirming, I'm glad we're converging too!

Could you please send me also the call invite for tomorrow? I hope to be able to join for at least part of the discussion; thanks in advance!

[stscijgbot-jp]

Comment by Sarah Kendrew on JIRA:

Short summary from the meeting we held on June 8th @ 4 pm. Present: Anton Koekemoer, Karl Gordon, Nikolay Nikolov, Everett Schlawin, Thomas Beatty, Sarah Kendrew.

• It is clear that the issues "introduced" by the photom step are related to pointing jitter and whether or not we attempt to track/correct this in the subsequent analysis. it is correct that when we want to correct for this in the extracted product, we have to "roll back" the photometric calibration, which is cumbersome and for that reason it could help to switch the step off by default.
• we should also bear in mind that not all TSOs are transiting exoplanet observations (though the majority are); there are some accepted proposals that observe different types of objects
• there is no zero-effort solution; switching the step off is "easy" in principle (from a coding p.o.v) but it is impactful and it will require additional testing/updating of testing infrastructure to ensure that the L3 steps are able to execute on different output product from Spec2. 
• with that in mind we agreed that the step should be kept on for now, and that the TSO WG should work on a ticket for the implementation of a jitter correction step that tweaks the WCS on a per-integration basis. This could use a cross-correlation algorithm or something else TBD. The TSO WG should also determine the prioritisation of such a step in the context of other open TSO pipeline work items, and propose an algorithm.

I hope you all feels this more or less covers what we discussed - feel free to add or correct!

[stscijgbot-jp]

Comment by Anton Koekemoer on JIRA:

thanks Sarah Kendrew for the great summary, that's very helpful!

For your 1st point, a quick note that the procedure for tracking/ correcting pointing jitter offsets in "subsequent analysis" is post-pipeline, thus it's impacted by the photom step.

Also, the proposed solution (in your last point) to insert a step to correct these offsets earlier in the pipeline (before the photom step) would have the benefit of producing pipeline products with corrected spectral WCS, potentially removing the need to correct these offsets post-pipeline.

To track this further, for the TSO WG to discuss the next steps for agreeing on an algorithm (and/or existing code) to measure the offsets, to propose to the Cal WG, could you open a new Jira ticket please (eg titled "TSO spectroscopic jitter offset WCS correction" or something along those lines)?

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Given that the most recent consensus is to NOT turn off the photom step by default, but rather fix the jitter offsets via an independent correction, I feel that this ticket can be closed, with a link to the relevant new ticket for the jitter correction. Agreed?

[stscijgbot-jp]

Comment by Karl Gordon on JIRA:

Sounds reasonable to me.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Temporarily reopening to get resolution set correctly to withdrawn.