Closed stscijgbot-jp closed 4 years ago
stscijgbot-jp
commented
4 years ago Comment by Howard Bushouse on JIRA:
The same result occurs for program 623, obs 20 and 31. Same situation with FGS in parallel to MIRI. image2 ASN files get created for all of the FGS guider2 exposures listed in the ASN pool, but no image3 ASN's get created. The pool file jw00623_20191210t195246_pool.csv has been copied into the test data directory mentioned above.
stscijgbot-jp
commented
4 years ago Comment by Shannon Osborne on JIRA:
This ticket has been marked as critical for the FGS team because it is required for commissioning. The justification from the FGS team is "If things are associated properly in DMS, it might make it easier to find & retrieve the data if/when needed. This could be especially important in time-crunch times like commissioning."
stscijgbot-jp
commented
4 years ago Comment by Howard Bushouse on JIRA:
The latest results of the investigation of this problem indicate that it's not actually a problem in the ASN generator rules or Cal pipeline, but is due to the fact the FGS PPSDB table entry for target_id is not populated for the case FGS parallel observations, and perhaps specifically FGS external calibration exposures taken as parallels. FGS science exposures should not be affected.
The solution is needed upstream, such as in APT/PPS, so that the target_id field gets populated with something.
stscijgbot-jp
commented
4 years ago Comment by Anton Koekemoer on JIRA:
Adding notes from a discussion Wed May 20, 2020 and also a long associated email thread.
Suggest that further discussion should continue as comments on this Jira ticket.
-------- Forwarded Message -------- Subject: Re: Question about FGS parallel observations Date: Wed, 20 May 2020 17:17:59 -0400 From: Anton M. Koekemoer koekemoer@stsci.edu To: Matthew D. Lallo lallo@stsci.edu, Pierre Chayer chayer@stsci.edu, Ed Nelan nelan@stsci.edu, Howard Bushouse bushouse@stsci.edu, Alicia Canipe acanipe@stsci.edu, Charles-Philippe Lajoie lajoie@stsci.edu, Shannon Osborne sosborne@stsci.edu, tsohn@stsci.edu, Sherie Holfeltz holfeltz@stsci.edu, Anton M. Koekemoer koekemoer@stsci.edu
hi all,
thanks very much for the discussion today, which was very helpful.
Here are my notes for a path forward, also adding Tony and Sherie to the email chain here as Matt suggested (the entire email trail is below, for context).
Also at this point I'll go ahead and post this all into the JIRA ticket JP-1409, and maybe future discussion can continue there. Also adding everyone on this email thread as watchers to that ticket.
Summary: this issue relates to parallel FGS observations not processing through to level-3 combination in the pipeline, because the target_id is set to Null somewhere within APT/PPS (instead of being populated, as it is for other instruments in parallel).
Note: prime observations with FGS (and other instruments) already process fully through the pipeline, producing "level 3" combined image products when the exposure type is IMAGE (but not when it's DARK or FLAT) so this is all fine; the issue here is only for parallel modes.
There's also a nomenclature issue in the email thread and in JP-1409 - some places refer to "level 3 ASNs" but for clarity, let's rather refer to them as "level 3 combined image products" (since "ASN" may simply refer to a list of files that are associated, and not the actual combined image product).
Alicia Canipe, Howard Bushouse, Pierre Chayer, Anton Koekemoer, Matt Lallo, Shannon Osborne
(1) What data products are needed from the DMS pipeline for FGS-related commissioning activities?
The two programs mentioned in this JIRA ticket JP-1409 (ie programs 623 and 624) are in fact SOR programs, not commissioning programs.
There are some FGS-related commissioning programs that use FGS in parallel, such as FGS-SI alignment, possibly also others. These are not all within the FGS team (eg NIRCam-FGS alignment is NRC-21, within NIRCam, on https://outerspace.stsci.edu/display/JN/NIRCam+Commissioning+Analysis+Plans+and+Reports0]), while more FGS-specific commissioning activities are on a separate page ([https://outerspace.stsci.edu/display/JWSTSC/FGS+Commissioning])
For commissioning activities related to FGS-SI alignment and astrometry, the general expectation is to use "level 2b" pipeline products, namely individual exposures, specifically without distortion correction applied. Matt Lallo mentioned a technical report by Johannes Sahlmann (JWST-STScI-007075, available in SOCCER) which describes the expected analysis procedure for these commissioning plans, and which specifically mentions the expectation of using "level 2b" (ie individual calibrated exposures, without distortion correction), and not "level 3" (which would be combined products).
There may however be other commissioning activities, with FGS in parallel, that may need combined level-3 data products. It may also be desirable to create these in general, beyond what's needed for commissioning (see later).
There is a page organized by Alicia/ Rossy in 2019, summarizing commissioning needs for the instruments ([https://outerspace.stsci.edu/display/JWSTPWG/Plans+for+Calibration+Pipeline+Usage+During+Commissioning]), however this page doesn't capture whether the modes are prime or parallel.
(2) Does the DMS pipeline currently produce level-3 combined image products in general, for parallel observations?
for other instruments (non-FGS) in parallel, the target_id is set, and this should enable them to process through to produce level-3 combined image products.
for FGS in parallel the target_id is set to Null somewhere within APT/PPS (instead of being populated, as it is for other instruments in parallel), and this causes them to not process through to produce level-3 combined image products.
(3) Do we want FGS level-3 products in general, for parallel observations?
Pierre mentioned there are potential use cases where this can be needed, for example measuring photometry or PSF properties on combined images.
Note here however that FGS in parallel is not supported for science, it's only available as an engineering template.
From the discussion today (and the email thread): no changes to the pipeline code appear to be needed (ie, the pipeline can and will process FGS imaging data through level-3 combination, if it has the right metadata), it's more a case of relevant flags not being set on the DMS side (eg, target_id set to Null).
If it's decided that having level-3 combined data for parallel FGS IMAGE exposures is useful, then this could be achieved by setting target_id to an appropriate value; the discussion now moves out from the Cal WG side and back to the DMS side.
Matthew D. Lallo (or someone he designates, eg Tony or Sherie) - reach out to the relevant people planning to work on FGS-related analysis during commissioning, when FGS observations are in parallel, to determine whether any of them need level-3 products (ie, combined images) from the DMS pipeline for their commissioning analysis, and if so, which program IDs are these?
Alicia Canipe (or someone she designates) - confirm what tests have been done so far on processing parallel modes through the pipeline end-to-end, in terms of the various prime+par instrument combinations (if there's a webpage summarizing this, that may be the best way to present this)
Howard Bushouse (or someone she designates) - verify with the relevant staff from DMD/PPS further down the email chain to see if anyone knows why there's a specific statement to explicitly set target_id to Null for these parallel FGS exposures; once we know the reason, this can inform the next steps.
thanks again, Anton.
On 5/20/20 13:54:11 , Matthew D. Lallo wrote:
Thanks Pierre (and Ed),
I will try to join the 2pm, but can say the following now.
When we use FGS in parallel with an SI for astrometric calibrations and focal plane alignments, we do not require the level 3 or even the level 2b products. The astrometric reduction is performed on the distorted sources prior to those pipeline steps. If we have higher-level products that’s fine, but we don’t use them so don’t require them, so this use case alone would not drive the pipeline question being pursued.
I think the question now is if FGS parallels would be used for other types of calibrations, like the photometric calibration Pierre describes, and does this benefit from level 3 processing?
This is the distinction being made. I am not immediately aware of the nature of FGS photometric calibrations during the science cycle, but during commissioning, when this template is first being utilized, its primary function is for astrometric/geometric characterization and calibration.
____ Matt Lallo, Mission Systems Scientist Lead, Telescopes Group Space Telescope Science Institute 3700 San Martin Drive Baltimore, MD 21218 Office: +1 410 338 4577 Mobile: +1 443 739 8425
On May 20, 2020, at 1:43 PM, Pierre Chayer <chayer@stsci.edu [chayer@stsci.edu|mailto:chayer@stsci.edu]> wrote:
Hi Ed,
I must say that I disagree with your assessment. Although these observations are part of the FGS-SI alignment calibration programs, they can be used to improve our FGS photometric calibration for guiding. I would suggest that we request the best data products that we can get to carry out and improve our photometric calibration as we go through commissioning.
Thanks.
Pierre
On 5/20/20, 12:53 PM, "Ed Nelan" <nelan@stsci.edu [nelan@stsci.edu|mailto:nelan@stsci.edu]> wrote:
Unfortunately I won't be able to attend the 2pm meeting. A quick read of the email trail and a look at the two proposals, 623,624, indicates to me that we don't need the level-3 products. These proposals are for FGS-SI alignment calibrations, so I think we will be using the level 2a & 2b products. That said, having a level-3 ASN could be convenient, as I understand, for extracting the (level 2?) observations as a group from MAST, but otherwise no other benefit. At any rate this is my understanding of the issue, which may be incomplete.
Thanks Ed
On 5/20/20, 11:28 AM, "Anton Koekemoer" <koekemoer@stsci.edu [koekemoer@stsci.edu|mailto:koekemoer@stsci.edu]> wrote:
hi all,
thanks Howard for the terminology clarifications. To be honest, I think at this point we do need a tagup to be sure we're all talking about the same thing, in order to decide how to proceed.
So I've taken the liberty of looking at everyone's calendar and scheduling a tagup for a time that seems to work, ie today at 2:00pm
I'll send the webex invite shortly
thanks again, - Anton.
On 5/20/20 11:10:52 , Howard Bushouse wrote:
OK, we just have a small terminology issue. I tend to use the terms "dither" and "mosaic" interchangeably, because they're all exposures that have different pointing positions on the sky, it's just that mosaics have bigger differences in pointing than dithers. Level-3 image combination applies equally to exposures that are dithered or mosaiced or both. We don't distinguish between the two in terms of whether or not to apply level-3 processing.
So now it sounds like you do need/want level-3 processing for the FGS_IMAGE exposures taken in parallel, regardless of whether they use dither or mosaic patterns, in which case we'll need to fix the missing target_id problem to make that happen.
-Howard
On 5/20/20, 11:04 AM, "Pierre Chayer" <chayer@stsci.edu [chayer@stsci.edu|mailto:chayer@stsci.edu]> wrote:
Hi Anton and Howard,
I may have misunderstood the level-3 processing of dithered FGS_IMAGE exposures. Is the combination of the dithered FGS_IMAGE exposures processed by the calwebb_image3 module? In my previous email I was declining the mosaic combination, but not the dithered exposures combination.
Thanks
Pierre
On 5/20/20, 10:28 AM, "Howard Bushouse" <bushouse@stsci.edu [bushouse@stsci.edu|mailto:bushouse@stsci.edu]> wrote:
Anton,
Just to be completely clear, the statement in JP-1409 "The dithered FGS exposures should result in image3 ASN's, for processing in the calwebb_image3 pipeline" was made by me (not the FGS team), under the assumption that any/all dithered exposures should go through level-3 processing. We've now learned that the FGS team does not need or desire level-3 processing, at least for this particular case of parallel external calibration exposures. So it sounds like we might be able to withdraw/reject this PR.
Howard
On 5/20/20, 10:14 AM, "Anton Koekemoer" <koekemoer@stsci.edu [koekemoer@stsci.edu|mailto:koekemoer@stsci.edu]> wrote:
hi Pierre,
ok, thanks for your reply. In that case, could you help us understand specifically what's needed for the ticket? Since it says in the description of JP-1409:
"The dithered FGS exposures should result in image3 ASN's, for processing in the calwebb_image3 pipeline."
So, if the FGS_IMAGE exposure-type products don't need to be processed through level-3 mosaic combination (for this program 624, also 623), then can your needs instead be satisfied with level-1 or level-2 products?
Basically we're trying to find out in a bit more detail what it is that the FGS team needs - if there's no need for actual pipeline products from image3 (ie, no need for combined mosaics), and if instead you'll do the commissioning analysis with level-1 or level-2 products, then we're trying to see why image3 ASNs are needed (and why not, eg level-1 or level-2 ASNs)
Let us know if you'd prefer to schedule a tagup to discuss this further and finalize it, if that would be helpful
thanks very much again, - Anton.
On 5/20/20 9:43:22 , Pierre Chayer wrote:
Hi Anton,
We are not requesting that the FGS_IMAGE exposure-type products be processed through level-3 mosaic combination.
Thanks.
Pierre
On 5/19/20, 10:32 PM, "Anton Koekemoer" <koekemoer@stsci.edu [koekemoer@stsci.edu|mailto:koekemoer@stsci.edu]> wrote:
hi Pierre,
thanks very much for confirming that you're not requesting level-3 products for darks and skyflats, so this does help to focus the discussion somewhat.
To clarify a bit further then, are you requesting that these FGS IMAGE exposure-type products be processed through level-3 mosaic combination, for thee programs (623 and 624) in order for you to be able to do your commissioning analysis? If that's the case then that would be helpful for the discussion, in order to determine the next steps forward.
thanks very much again, best regards, - Anton.
On 5/19/20 18:13:20 , Pierre Chayer wrote:
Dear Alicia, Howard, and Anton,
Thank you for describing the issue regarding the FGS level-3 products. I greatly appreciate your efforts to resolve the issue.
We do not request level-3 products for DARK and SKYFLAT images.
The FGS_IMAGE exposure type is indeed normal on-sky exposure. This IMAGE exposure type is similar to the other science instruments IMAGE exposure type.
I'm not sure how to move forward with this issue. It seems that we are stuck with the target_id that is intentionaly not populated in the PPSDB table for the parallel FGS exposures. I am open to suggestions.
Thanks.
Pierre
On 5/19/20, 4:36 PM, "Anton Koekemoer" <koekemoer@stsci.edu [koekemoer@stsci.edu|mailto:koekemoer@stsci.edu]> wrote:
hi all,
thanks Alicia for the summary, I agree with Howard this captures it nicely.
And the main reason now for circling back to the FGS team is to try to understand what specifically the needs are that motivate JP-1409, if we could have some more clarification from FGS on the following:
1) is it a need to obtain actual level-3 products, eg a combined mosaic, of these parallel FGS exposures, and if so, what type of exposures are these? (level-3 combined mosaics are likely not very useful if these are darks or flats, which are also not passed through to level-3 combination when taken as prime for other instruments)
1) if instead the need for level-3 ASNs is just to make it easier to do data retrieval from AST, which the ticket seems to suggest, ie just for individual exposures and not combined mosaics, then could you help clarify why it's the level-3 ASNs that are needed (and not eg level-1) and what processing you need to have done to them, beyond level-1? (especially if they're indeed darks or flats)
thanks very much again, - Anton.
On 5/19/20 15:47:11 , Howard Bushouse wrote:
You covered the details pretty well, Alicia. The main problem is the fact that target_id is intentionally not populated in the PPSDB table for the parallel FGS exposures and no one in APT can recall right now why that decision was made. And a taget_id value is critical for forming associations, since the #1 selection criterion is that exposures have the same target_id in order to be associated (and a null value doesn’t count). So there’s no way right now for level-3 ASN’s to get created for the parallel FGS external calibration exposures.
Howard
From: Alicia Canipe <acanipe@stsci.edu [acanipe@stsci.edu|mailto:acanipe@stsci.edu]> Date: Tuesday, May 19, 2020 at 1:56 PM To: Pierre Chayer <chayer@stsci.edu [chayer@stsci.edu|mailto:chayer@stsci.edu]>, Ed Nelan <nelan@stsci.edu [nelan@stsci.edu|mailto:nelan@stsci.edu]>, "Matthew D. Lallo" <lallo@stsci.edu [lallo@stsci.edu|mailto:lallo@stsci.edu]>, Charles-Philippe Lajoie <lajoie@stsci.edu [lajoie@stsci.edu|mailto:lajoie@stsci.edu]>, Shannon Osborne <sosborne@stsci.edu [sosborne@stsci.edu|mailto:sosborne@stsci.edu]> Cc: Anton Koekemoer <koekemoer@stsci.edu [koekemoer@stsci.edu|mailto:koekemoer@stsci.edu]>, Howard Bushouse <bushouse@stsci.edu [bushouse@stsci.edu|mailto:bushouse@stsci.edu]> Subject: Question about FGS parallel observations
Hi everyone,
There has been some discussion about how to process FGS external calibration exposures, motivated by this ticket: https://jira.stsci.edu/browse/JP-1409. Can you clarify the expectations of the FGS team for the following (roughly extracted from the long thread copied below):
For the case of FGS parallel calibration exposures, like DARK and SKYFLAT for FGS external calibrations, no level-2b or level-3 processing should be done at all for darks and flats (i.e., they are processed the same way they would be in prime), correct? In the FGS IMAGE case for parallel calibration exposures (which appear to be normal on-sky exposures), does DMS need to be set up to do full level-2b and level-3 processing? In the JP-1409 issue, it seems like the generation of association products needed for level-3 processing failed because this is a parallel FGS external calibration observation, so even if the assigned exposure type is FGS_IMAGE, target ID does not get populated for pure and engineering parallels, so DMS cannot generate the association needed to associate the exposures correctly. They would need to make some changes in order to process parallel FGS_IMAGE calibration exposures beyond level-2a, if that is what is needed.
Let us know your thoughts and what you all are expecting for these types of exposures. I’ll copy this conversation into JP-1409 for background once we have consensus from the FGS experts. (Howard or Anton, feel free to comment if I didn’t quite capture the question from DMS).
Thanks for your time,
Alicia
Background thread:
From: Howard Bushouse <bushouse@stsci.edu [bushouse@stsci.edu|mailto:bushouse@stsci.edu]> Date: Tuesday, May 19, 2020 at 7:31 AM To: Mark Kyprianou <kyp@stsci.edu [kyp@stsci.edu|mailto:kyp@stsci.edu]>, Daryl Swade <swade@stsci.edu [swade@stsci.edu|mailto:swade@stsci.edu]>, Jeff Valenti <valenti@stsci.edu [valenti@stsci.edu|mailto:valenti@stsci.edu]> Cc: Andrew Myers <amyers@stsci.edu [amyers@stsci.edu|mailto:amyers@stsci.edu]>, Rob Douglas <rdouglas@stsci.edu [rdouglas@stsci.edu|mailto:rdouglas@stsci.edu]>, Jonathan Eisenhamer <eisenhamer@stsci.edu [eisenhamer@stsci.edu|mailto:eisenhamer@stsci.edu]>, Robert Hawkins <rhawkins@stsci.edu [rhawkins@stsci.edu|mailto:rhawkins@stsci.edu]>, "Richard A. Shaw" <shaw@stsci.edu [shaw@stsci.edu|mailto:shaw@stsci.edu]>, David Davis <ddavis@stsci.edu [ddavis@stsci.edu|mailto:ddavis@stsci.edu]>, jwst-dms-dev-leads <jwst-dms-dev-leads@stsci.edu [jwst-dms-dev-leads@stsci.edu|mailto:jwst-dms-dev-leads@stsci.edu]> Subject: Re: FGS Parallel Observations
This question is best posed to INS, since it’s their expectations that are in question. But personally, for the particular case of parallel calibration exposures, like the DARK and SKYFLAT possibilities mentioned in this thread for FGS external calibrations, I would say no. No level-2b or level-3 processing should be done at all for darks and flats (it isn’t for darks and flats that are prime) and even if level-2b processing were applied, there’s no point in doing level-3. You don’t want flats built into a mosaic. They’re intended to be stacked/combined in detector space. For the FGS “IMAGE” mode, on the other hand, which appear to be normal on-sky exposures, then perhaps full level-2b and level-3 processing is warranted and desired.
Howard
From: Mark Kyprianou <kyp@stsci.edu [kyp@stsci.edu|mailto:kyp@stsci.edu]> Date: Monday, May 18, 2020 at 6:37 PM To: Daryl Swade <swade@stsci.edu [swade@stsci.edu|mailto:swade@stsci.edu]>, Jeff Valenti <valenti@stsci.edu [valenti@stsci.edu|mailto:valenti@stsci.edu]> Cc: Andrew Myers <amyers@stsci.edu [amyers@stsci.edu|mailto:amyers@stsci.edu]>, Rob Douglas <rdouglas@stsci.edu [rdouglas@stsci.edu|mailto:rdouglas@stsci.edu]>, Jonathan Eisenhamer <eisenhamer@stsci.edu [eisenhamer@stsci.edu|mailto:eisenhamer@stsci.edu]>, Robert Hawkins <rhawkins@stsci.edu [rhawkins@stsci.edu|mailto:rhawkins@stsci.edu]>, Howard Bushouse <bushouse@stsci.edu [bushouse@stsci.edu|mailto:bushouse@stsci.edu]>, "Richard A. Shaw" <shaw@stsci.edu [shaw@stsci.edu|mailto:shaw@stsci.edu]>, David Davis <ddavis@stsci.edu [ddavis@stsci.edu|mailto:ddavis@stsci.edu]>, jwst-dms-dev-leads <jwst-dms-dev-leads@stsci.edu [jwst-dms-dev-leads@stsci.edu|mailto:jwst-dms-dev-leads@stsci.edu]> Subject: Re: FGS Parallel Observations
This discussion started as a result of INS expecting association products for FGS external calibrations, so at least in one case, they were expecting them… is this a common expectation for coordinated parallels?
Regards, mark
From: Daryl Swade <swade@stsci.edu [swade@stsci.edu|mailto:swade@stsci.edu]> Date: Monday, May 18, 2020 at 5:47 PM To: Robert Hawkins <rhawkins@stsci.edu [rhawkins@stsci.edu|mailto:rhawkins@stsci.edu]> Cc: Andrew Myers <amyers@stsci.edu [amyers@stsci.edu|mailto:amyers@stsci.edu]>, Mark Kyprianou <kyp@stsci.edu [kyp@stsci.edu|mailto:kyp@stsci.edu]>, Rob Douglas <rdouglas@stsci.edu [rdouglas@stsci.edu|mailto:rdouglas@stsci.edu]>, Jeff Valenti <valenti@stsci.edu [valenti@stsci.edu|mailto:valenti@stsci.edu]>, Jonathan Eisenhamer <eisenhamer@stsci.edu [eisenhamer@stsci.edu|mailto:eisenhamer@stsci.edu]>, Howard Bushouse <bushouse@stsci.edu [bushouse@stsci.edu|mailto:bushouse@stsci.edu]>, "Richard A. Shaw" <shaw@stsci.edu [shaw@stsci.edu|mailto:shaw@stsci.edu]>, David Davis <ddavis@stsci.edu [ddavis@stsci.edu|mailto:ddavis@stsci.edu]>, jwst-dms-dev-leads <jwst-dms-dev-leads@stsci.edu [jwst-dms-dev-leads@stsci.edu|mailto:jwst-dms-dev-leads@stsci.edu]> Subject: Re: FGS Parallel Observations
We will leave it to DMS to decide if they want to peruse this effort. If so, a JSOCINT ticket should be filed to coordinate and make everyone aware of the change. Agreed, this is only for coordinated parallels.
Thanks,
Daryl
From: Robert Hawkins <rhawkins@stsci.edu [rhawkins@stsci.edu|mailto:rhawkins@stsci.edu]> Date: Monday, May 18, 2020 at 5:39 PM To: Daryl Swade <swade@stsci.edu [swade@stsci.edu|mailto:swade@stsci.edu]> Cc: Andrew Myers <amyers@stsci.edu [amyers@stsci.edu|mailto:amyers@stsci.edu]>, Mark Kyprianou <kyp@stsci.edu [kyp@stsci.edu|mailto:kyp@stsci.edu]>, Rob Douglas <rdouglas@stsci.edu [rdouglas@stsci.edu|mailto:rdouglas@stsci.edu]>, Jeff Valenti <valenti@stsci.edu [valenti@stsci.edu|mailto:valenti@stsci.edu]>, Jonathan Eisenhamer <eisenhamer@stsci.edu [eisenhamer@stsci.edu|mailto:eisenhamer@stsci.edu]>, Howard Bushouse <bushouse@stsci.edu [bushouse@stsci.edu|mailto:bushouse@stsci.edu]>, "Richard A. Shaw" <shaw@stsci.edu [shaw@stsci.edu|mailto:shaw@stsci.edu]>, David Davis <ddavis@stsci.edu [ddavis@stsci.edu|mailto:ddavis@stsci.edu]>, jwst-dms-dev-leads <jwst-dms-dev-leads@stsci.edu [jwst-dms-dev-leads@stsci.edu|mailto:jwst-dms-dev-leads@stsci.edu]> Subject: Re: FGS Parallel Observations
Daryl
While it’s doable (and easy) for us to populate the target_id for coordinated parallels only, that doesn’t do anything for the other situations (non-coordinated engineering, and pure parallels). Also, as Andrew pointed out, strictly speaking, the prime target is not the “target" of the coordinated parallel.
We should be certain that no other system would care if we do populate for coordinated as we explicitly chose to not do so in the initial implementation. Several other systems read the exposures table and we wouldn’t want unexpected side effects.
This implementation was documented in the initial implementing ticket for coordinated parallels (https://jira.stsci.edu/browse/APT-82541) but no reasons were recorded.
Rob
On May 18, 2020, at 5:22 PM, Daryl Swade <swade@stsci.edu [swade@stsci.edu|mailto:swade@stsci.edu] [swade@stsci.edu|mailto:swade@stsci.edu]> wrote:
I did say the target id was populated in the exposures table for coordinated parallels, but I didn’t mean it. Sorry for the confusion.
The target id is not populated for the parallel exposures in coordinated parallels, but it would be helpful to DMS if it was populated. The target is specified at the observation level, so the same target id as the prime could be used for the coordinated parallel. Based on Howard’s comments, DMS uses the target to form associations and these coordinated parallel observations can execute dithers and mosaics, even for engineering templates.
Here are some specifics. Looking through APT I find the following coordinated parallel combinations in engineering proposal mode:
NIRSpec MOS - NIRCam Imaging
NIRSpec MOS - MIRI Imaging
MIRI imaging - NIRCam Imaging
MIRI Imaging - NIRISS WFSS
NIRCam Imaging - MIRI Imaging
NIRCam Imaging - NIRISS Imaging
NIRCam Imaging - NIRISS WFSS
NIRCam WFSS - MIRI Imaging
NIRCam WFSS - NIRISS Imaging
NIRCam Eng Imaging - FGS Ext Cal
NIRISS Ext Cal - FGS Ext Cal
The last two are engineering templates. However, dithers and mosaics are possible in these two cases, and DMS needs information to generate associations. In all cases, the target appears at the observation level.
Here is information inserted into the exposures table from the APT .sql output for a few examples. In APT I placed a fixed target in the observation.
FGS only: target_id = 1
MIRI imaging - NIRCam Imaging
MIRI exposure: target_id = 1
NIRCam exposure: target_id not in the insert into exposures statement
NIRCam Eng Imaging - FGS Ext Cal
NIRCam exposure: target_id = 1
FGS exposure: target_id not in the insert into exposures statement
Same result for FGS exposure target type = DARK, SKYFLAT, and IMAGE.
Daryl
From:Robert Hawkins <rhawkins@stsci.edu [rhawkins@stsci.edu|mailto:rhawkins@stsci.edu] [rhawkins@stsci.edu|mailto:rhawkins@stsci.edu]> Date:Monday, May 18, 2020 at 3:11 PM To:Andrew Myers <amyers@stsci.edu [amyers@stsci.edu|mailto:amyers@stsci.edu] [amyers@stsci.edu|mailto:amyers@stsci.edu]> Cc:Mark Kyprianou <kyp@stsci.edu [kyp@stsci.edu|mailto:kyp@stsci.edu] [kyp@stsci.edu|mailto:kyp@stsci.edu]>, Rob Douglas <rdouglas@stsci.edu [rdouglas@stsci.edu|mailto:rdouglas@stsci.edu] [rdouglas@stsci.edu|mailto:rdouglas@stsci.edu]>, Daryl Swade <swade@stsci.edu [swade@stsci.edu|mailto:swade@stsci.edu] [swade@stsci.edu|mailto:swade@stsci.edu]>, Jeff Valenti <valenti@stsci.edu [valenti@stsci.edu|mailto:valenti@stsci.edu] [valenti@stsci.edu|mailto:valenti@stsci.edu]>, Jonathan Eisenhamer <eisenhamer@stsci.edu [eisenhamer@stsci.edu|mailto:eisenhamer@stsci.edu] [eisenhamer@stsci.edu|mailto:eisenhamer@stsci.edu]>, Howard Bushouse <bushouse@stsci.edu [bushouse@stsci.edu|mailto:bushouse@stsci.edu] [bushouse@stsci.edu|mailto:bushouse@stsci.edu]>, "Richard A. Shaw" <shaw@stsci.edu [shaw@stsci.edu|mailto:shaw@stsci.edu] [shaw@stsci.edu|mailto:shaw@stsci.edu]>, David Davis <ddavis@stsci.edu [ddavis@stsci.edu|mailto:ddavis@stsci.edu] [ddavis@stsci.edu|mailto:ddavis@stsci.edu]>, jwst-dms-dev-leads <jwst-dms-dev-leads@stsci.edu [jwst-dms-dev-leads@stsci.edu|mailto:jwst-dms-dev-leads@stsci.edu] [jwst-dms-dev-leads@stsci.edu|mailto:jwst-dms-dev-leads@stsci.edu]> Subject:Re: FGS Parallel Observations
Just to be clear, though it looked like Daryl was saying we do populate for Coordinated, whet I see in the code (as shown below) indicates that we don’t, and when I looked at the SQL, that was confirmed in the examples I had.
Rob
On May 18, 2020, at 3:10 PM, Andrew Myers <amyers@stsci.edu [amyers@stsci.edu|mailto:amyers@stsci.edu] [amyers@stsci.edu|mailto:amyers@stsci.edu]> wrote:
I don’t have much more to add, I think we would need requirements here.
As Rob H. said, APT has no idea what a pure parallel will be in parallel with, so we couldn’t do any more than a generic tag. I’m surprised we populate anything for coordinated parallels. We also don’t know what part of the sky the coordinated parallel will be pointing at because we don’t know what the orient will be. I suppose using the prime target name is better than nothing? But it’s not what’s under the instrument field of view.
Andrew
From:Robert Hawkins <rhawkins@stsci.edu [rhawkins@stsci.edu|mailto:rhawkins@stsci.edu] [rhawkins@stsci.edu|mailto:rhawkins@stsci.edu]> Date:Friday, May 15, 2020 at 17:26 To:Mark Kyprianou <kyp@stsci.edu [kyp@stsci.edu|mailto:kyp@stsci.edu] [kyp@stsci.edu|mailto:kyp@stsci.edu]> Cc:Rob Douglas <rdouglas@stsci.edu [rdouglas@stsci.edu|mailto:rdouglas@stsci.edu] [rdouglas@stsci.edu|mailto:rdouglas@stsci.edu]>, Andrew Myers <amyers@stsci.edu [amyers@stsci.edu|mailto:amyers@stsci.edu] [amyers@stsci.edu|mailto:amyers@stsci.edu]>, Daryl Swade <swade@stsci.edu [swade@stsci.edu|mailto:swade@stsci.edu] [swade@stsci.edu|mailto:swade@stsci.edu]>, Jeff Valenti <valenti@stsci.edu [valenti@stsci.edu|mailto:valenti@stsci.edu] [valenti@stsci.edu|mailto:valenti@stsci.edu]>, Jonathan Eisenhamer <eisenhamer@stsci.edu [eisenhamer@stsci.edu|mailto:eisenhamer@stsci.edu] [eisenhamer@stsci.edu|mailto:eisenhamer@stsci.edu]>, Howard Bushouse <bushouse@stsci.edu [bushouse@stsci.edu|mailto:bushouse@stsci.edu] [bushouse@stsci.edu|mailto:bushouse@stsci.edu]>, "Richard A. Shaw" <shaw@stsci.edu [shaw@stsci.edu|mailto:shaw@stsci.edu] [shaw@stsci.edu|mailto:shaw@stsci.edu]>, David Davis <ddavis@stsci.edu [ddavis@stsci.edu|mailto:ddavis@stsci.edu] [ddavis@stsci.edu|mailto:ddavis@stsci.edu]>, jwst-dms-dev-leads <jwst-dms-dev-leads@stsci.edu [jwst-dms-dev-leads@stsci.edu|mailto:jwst-dms-dev-leads@stsci.edu] [jwst-dms-dev-leads@stsci.edu|mailto:jwst-dms-dev-leads@stsci.edu]> Subject:Re: FGS Parallel Observations
Some APT-specific (mostly) responses.Rob or Andrew may have more to contribute from a general PPS perspective:
There are really 3 types of parallels: Pure, Coordinated and Engineering (the latter is defined as engineering template proposals, including FGS, which have the “Parallel” SR attached). I think there’s confusion throughout this thread on these.
For the first case, coordinated parallels, APT would know the target (from the prime), but only the prime gets the target_id populated in the exposures table, by design:
/// Non-prime coordinated parallel exposures should not populate target_id. /boolean lWriteTargetId = true; if(iPointing.getExposure() != null && iPointing.getExposure().getTemplate().isCoordinatedParallelAndNonPrime())
{ >>>>>> lWriteTargetId = false; >>>>>> }
For pure and engineering proposals, as Howard alluded, APT would not be able to specify the target ID as there’s no associated target. (For Pure Parallel proposals, there’s not even a target folder in the proposal). For engineering parallels, likewise, target is specified to be NONE, so there’s no target_id to export.
I think that we’d have to populate this downstream, in vss(?) wherever attachment occurs.
Rob
On May 15, 2020, at 4:39 PM, Mark Kyprianou <kyp@stsci.edu [kyp@stsci.edu|mailto:kyp@stsci.edu] [kyp@stsci.edu|mailto:kyp@stsci.edu]> wrote:
Adding in PPS folks to this thread, to get their feedback…. Depending on the PPS feedback, I think we’re about to the point where this moves to a JSOCINT ticket…
Summary:
·Parallel FGS observations are missing target_id
oIs this by design or a bug?
·Do all other parallel instrument combinations contain a specification of the target_id
oIt appears that most parallel cases do have the target_id specified
NOTE: the target_id field is needed for DMS association processing.
Regards, mark
From:Daryl Swade <swade@stsci.edu [swade@stsci.edu|mailto:swade@stsci.edu] [swade@stsci.edu|mailto:swade@stsci.edu]> Date:Thursday, May 14, 2020 at 1:51 PM To:Howard Bushouse <bushouse@stsci.edu [bushouse@stsci.edu|mailto:bushouse@stsci.edu] [bushouse@stsci.edu|mailto:bushouse@stsci.edu]>, Mark Kyprianou <kyp@stsci.edu [kyp@stsci.edu|mailto:kyp@stsci.edu] [kyp@stsci.edu|mailto:kyp@stsci.edu]>, Jeff Valenti <valenti@stsci.edu [valenti@stsci.edu|mailto:valenti@stsci.edu] [valenti@stsci.edu|mailto:valenti@stsci.edu]> Cc:Jonathan Eisenhamer <eisenhamer@stsci.edu [eisenhamer@stsci.edu|mailto:eisenhamer@stsci.edu] [eisenhamer@stsci.edu|mailto:eisenhamer@stsci.edu]>, "Richard A. Shaw" <shaw@stsci.edu [shaw@stsci.edu|mailto:shaw@stsci.edu] [shaw@stsci.edu|mailto:shaw@stsci.edu]>, David Davis <ddavis@stsci.edu [ddavis@stsci.edu|mailto:ddavis@stsci.edu] [ddavis@stsci.edu|mailto:ddavis@stsci.edu]>, jwst-dms-dev-leads <jwst-dms-dev-leads@stsci.edu [jwst-dms-dev-leads@stsci.edu|mailto:jwst-dms-dev-leads@stsci.edu] [jwst-dms-dev-leads@stsci.edu|mailto:jwst-dms-dev-leads@stsci.edu]> Subject:Re: FGS Parallel Observations
At least in the one MIRI-NIRCam example I looked at, both the MIRI and NIRCam exposures tables were populated.
Daryl
From:Howard Bushouse <bushouse@stsci.edu [bushouse@stsci.edu|mailto:bushouse@stsci.edu] [bushouse@stsci.edu|mailto:bushouse@stsci.edu]> Date:Thursday, May 14, 2020 at 1:32 PM To:Daryl Swade <swade@stsci.edu [swade@stsci.edu|mailto:swade@stsci.edu] [swade@stsci.edu|mailto:swade@stsci.edu]>, Mark Kyprianou <kyp@stsci.edu [kyp@stsci.edu|mailto:kyp@stsci.edu] [kyp@stsci.edu|mailto:kyp@stsci.edu]>, Jeff Valenti <valenti@stsci.edu [valenti@stsci.edu|mailto:valenti@stsci.edu] [valenti@stsci.edu|mailto:valenti@stsci.edu]> Cc:Jonathan Eisenhamer <eisenhamer@stsci.edu [eisenhamer@stsci.edu|mailto:eisenhamer@stsci.edu] [eisenhamer@stsci.edu|mailto:eisenhamer@stsci.edu]>, "Richard A. Shaw" <shaw@stsci.edu [shaw@stsci.edu|mailto:shaw@stsci.edu] [shaw@stsci.edu|mailto:shaw@stsci.edu]>, David Davis <ddavis@stsci.edu [ddavis@stsci.edu|mailto:ddavis@stsci.edu] [ddavis@stsci.edu|mailto:ddavis@stsci.edu]>, jwst-dms-dev-leads <jwst-dms-dev-leads@stsci.edu [jwst-dms-dev-leads@stsci.edu|mailto:jwst-dms-dev-leads@stsci.edu] [jwst-dms-dev-leads@stsci.edu|mailto:jwst-dms-dev-leads@stsci.edu]> Subject:Re: FGS Parallel Observations
Do we know if target_id gets filled in in PPSDB tables for other instruments used in parallel (e.g. NIRCam, MIRI, etc.)? If it’s just FGS external calibration exposures, it’s not as big of a deal. But if NIRCam or MIRI science parallels suffer from the same missing target_id, that’s a real issue.
Howard
From:Daryl Swade <swade@stsci.edu [swade@stsci.edu|mailto:swade@stsci.edu] [swade@stsci.edu|mailto:swade@stsci.edu]> Date:Thursday, May 14, 2020 at 1:24 PM To:Mark Kyprianou <kyp@stsci.edu [kyp@stsci.edu|mailto:kyp@stsci.edu] [kyp@stsci.edu|mailto:kyp@stsci.edu]>, Jeff Valenti <valenti@stsci.edu [valenti@stsci.edu|mailto:valenti@stsci.edu] [valenti@stsci.edu|mailto:valenti@stsci.edu]> Cc:Jonathan Eisenhamer <eisenhamer@stsci.edu [eisenhamer@stsci.edu|mailto:eisenhamer@stsci.edu] [eisenhamer@stsci.edu|mailto:eisenhamer@stsci.edu]>, "Richard A. Shaw" <shaw@stsci.edu [shaw@stsci.edu|mailto:shaw@stsci.edu] [shaw@stsci.edu|mailto:shaw@stsci.edu]>, Howard Bushouse <bushouse@stsci.edu [bushouse@stsci.edu|mailto:bushouse@stsci.edu] [bushouse@stsci.edu|mailto:bushouse@stsci.edu]>, David Davis <ddavis@stsci.edu [ddavis@stsci.edu|mailto:ddavis@stsci.edu] [ddavis@stsci.edu|mailto:ddavis@stsci.edu]>, jwst-dms-dev-leads <jwst-dms-dev-leads@stsci.edu [jwst-dms-dev-leads@stsci.edu|mailto:jwst-dms-dev-leads@stsci.edu] [jwst-dms-dev-leads@stsci.edu|mailto:jwst-dms-dev-leads@stsci.edu]> Subject:Re: FGS Parallel Observations
And for the FGS External Calibration exposures not in parallel, the target_id does get entered into the exposures table.
From:Daryl Swade <swade@stsci.edu [swade@stsci.edu|mailto:swade@stsci.edu] [swade@stsci.edu|mailto:swade@stsci.edu]> Date:Thursday, May 14, 2020 at 12:46 PM To:Mark Kyprianou <kyp@stsci.edu [kyp@stsci.edu|mailto:kyp@stsci.edu] [kyp@stsci.edu|mailto:kyp@stsci.edu]>, Jeff Valenti <valenti@stsci.edu [valenti@stsci.edu|mailto:valenti@stsci.edu] [valenti@stsci.edu|mailto:valenti@stsci.edu]> Cc:Jonathan Eisenhamer <eisenhamer@stsci.edu [eisenhamer@stsci.edu|mailto:eisenhamer@stsci.edu] [eisenhamer@stsci.edu|mailto:eisenhamer@stsci.edu]>, "Richard A. Shaw" <shaw@stsci.edu [shaw@stsci.edu|mailto:shaw@stsci.edu] [shaw@stsci.edu|mailto:shaw@stsci.edu]>, Howard Bushouse <bushouse@stsci.edu [bushouse@stsci.edu|mailto:bushouse@stsci.edu] [bushouse@stsci.edu|mailto:bushouse@stsci.edu]>, David Davis <ddavis@stsci.edu [ddavis@stsci.edu|mailto:ddavis@stsci.edu] [ddavis@stsci.edu|mailto:ddavis@stsci.edu]>, jwst-dms-dev-leads <jwst-dms-dev-leads@stsci.edu [jwst-dms-dev-leads@stsci.edu|mailto:jwst-dms-dev-leads@stsci.edu] [jwst-dms-dev-leads@stsci.edu|mailto:jwst-dms-dev-leads@stsci.edu]> Subject:Re: FGS Parallel Observations
The target_id is not being entered into the PPS exposures table for FGS as a coordinated parallel. Not sure if this is a bug or a feature? We should ask PPS.
Based on APT, FGS as a coordinated parallel is only allowed for engineering proposals. The three possible combinations for FGS as a coordinated parallel instrument are:
NIRCam Eng Imaging - FGS Ext Cal, can dither and mosaic
NIRISS Ext Cal - FGS Ext Cal, can dither and mosaic
NIRSpec Imaging - FGS Ext Cal, mosaic, no dither
For FGS External Calibration the three possible values for target type: IMAGE, DARK, SKYFLAT.
For a coordinated parallel observation, there is a single target specified in APT at the observations level.
I looked at a coordinated parallel observation with MIRI Imaging as prime and NIRCam Imaging in parallel. In this case the target_id was entered into the exposures table for both MIRI and NIRCam exposures.
Next, I looked at a coordinated parallel observation with NIRCam Engineering Imaging as prime and FGS External Calibration – SKYFLAT in parallel. Again, the target is specified at the observation level. However, in this case the target_id was entered into the exposures table for the NIRCam exposures, but not for the FGS exposures. The target_id was not entered into the exposures table for FGS IMAGE and DARK target types as well.
Daryl
From:Mark Kyprianou <kyp@stsci.edu [kyp@stsci.edu|mailto:kyp@stsci.edu] [kyp@stsci.edu|mailto:kyp@stsci.edu]> Date:Thursday, May 14, 2020 at 10:10 AM To:Jeff Valenti <valenti@stsci.edu [valenti@stsci.edu|mailto:valenti@stsci.edu] [valenti@stsci.edu|mailto:valenti@stsci.edu]>, Daryl Swade <swade@stsci.edu [swade@stsci.edu|mailto:swade@stsci.edu] [swade@stsci.edu|mailto:swade@stsci.edu]> Cc:Jonathan Eisenhamer <eisenhamer@stsci.edu [eisenhamer@stsci.edu|mailto:eisenhamer@stsci.edu] [eisenhamer@stsci.edu|mailto:eisenhamer@stsci.edu]>, "Richard A. Shaw" <shaw@stsci.edu [shaw@stsci.edu|mailto:shaw@stsci.edu] [shaw@stsci.edu|mailto:shaw@stsci.edu]>, Howard Bushouse <bushouse@stsci.edu [bushouse@stsci.edu|mailto:bushouse@stsci.edu] [bushouse@stsci.edu|mailto:bushouse@stsci.edu]>, David Davis <ddavis@stsci.edu [ddavis@stsci.edu|mailto:ddavis@stsci.edu] [ddavis@stsci.edu|mailto:ddavis@stsci.edu]>, jwst-dms-dev-leads <jwst-dms-dev-leads@stsci.edu [jwst-dms-dev-leads@stsci.edu|mailto:jwst-dms-dev-leads@stsci.edu] [jwst-dms-dev-leads@stsci.edu|mailto:jwst-dms-dev-leads@stsci.edu]> Subject:Re: FGS Parallel Observations
Adding in Jeff and Daryl to get their take on this email thread… We’ll file JSSET and/or JSOCINT tickets as appropriate, depending on their feedback.
Thanx for the investigation/suggestions!
Regards, mark
From:Howard Bushouse <bushouse@stsci.edu [bushouse@stsci.edu|mailto:bushouse@stsci.edu] [bushouse@stsci.edu|mailto:bushouse@stsci.edu]> Date:Thursday, May 14, 2020 at 9:25 AM To:"Richard A. Shaw" <shaw@stsci.edu [shaw@stsci.edu|mailto:shaw@stsci.edu] [shaw@stsci.edu|mailto:shaw@stsci.edu]>, David Davis <ddavis@stsci.edu [ddavis@stsci.edu|mailto:ddavis@stsci.edu] [ddavis@stsci.edu|mailto:ddavis@stsci.edu]> Cc:Mark Kyprianou <kyp@stsci.edu [kyp@stsci.edu|mailto:kyp@stsci.edu] [kyp@stsci.edu|mailto:kyp@stsci.edu]>, Jonathan Eisenhamer <eisenhamer@stsci.edu [eisenhamer@stsci.edu|mailto:eisenhamer@stsci.edu] [eisenhamer@stsci.edu|mailto:eisenhamer@stsci.edu]> Subject:Re: FGS Parallel Observations
I like your suggestions for possible target name syntax. So I wonder if it’s time to bump this issue up to the JSOCINT level, or whatever higher level is appropriate, because it should (I think) have buy-in from the instrument teams? A change really needs to be made somewhere upstream from DMS (e.g. either in APT or PPS?) in order to start populating target names for parallels. I’m not sure if it’s feasible to have SDP try to fill in a name, because we process one exposure at a time and hence when processing the parallel exposures, we’d have no info about the corresponding prime exposure(s) that it maps to. Or is there a way to do that?
Howard
From:"Richard A. Shaw" <shaw@stsci.edu [shaw@stsci.edu|mailto:shaw@stsci.edu] [shaw@stsci.edu|mailto:shaw@stsci.edu]> Date:Thursday, May 14, 2020 at 9:21 AM To:Howard Bushouse <bushouse@stsci.edu [bushouse@stsci.edu|mailto:bushouse@stsci.edu] [bushouse@stsci.edu|mailto:bushouse@stsci.edu]>, David Davis <ddavis@stsci.edu [ddavis@stsci.edu|mailto:ddavis@stsci.edu] [ddavis@stsci.edu|mailto:ddavis@stsci.edu]> Cc:Mark Kyprianou <kyp@stsci.edu [kyp@stsci.edu|mailto:kyp@stsci.edu] [kyp@stsci.edu|mailto:kyp@stsci.edu]>, Jonathan Eisenhamer <eisenhamer@stsci.edu [eisenhamer@stsci.edu|mailto:eisenhamer@stsci.edu] [eisenhamer@stsci.edu|mailto:eisenhamer@stsci.edu]> Subject:Re: FGS Parallel Observations
If something can be done to set a meaningful target name (and set the filter to “OPEN”) would allow creating meaningful associations, I’m certainly all for that. It’s not a supported science mode in Cy-1, but if CAL can do something useful, great. I would suggest a target name as some variant of the that for the prime instrument (‘NGC 1976 - FGS parallel"), rather than re-use some generic name, if only to avoid the possibility of someone searching MAST for FGS observations, only to find that the target “Parallel 1” matches coordinates all over the sky. (Also, a search by target name “NGC 1976*” would pick up both the prime and the FGS parallels.)
-Dick
From:Howard Bushouse <bushouse@stsci.edu [bushouse@stsci.edu|mailto:bushouse@stsci.edu] [bushouse@stsci.edu|mailto:bushouse@stsci.edu]> Date:Thursday, May 14, 2020 at 9:00 AM To:"Richard A. Shaw" <shaw@stsci.edu [shaw@stsci.edu|mailto:shaw@stsci.edu] [shaw@stsci.edu|mailto:shaw@stsci.edu]>, David Davis <ddavis@stsci.edu [ddavis@stsci.edu|mailto:ddavis@stsci.edu] [ddavis@stsci.edu|mailto:ddavis@stsci.edu]> Cc:Mark Kyprianou <kyp@stsci.edu [kyp@stsci.edu|mailto:kyp@stsci.edu] [kyp@stsci.edu|mailto:kyp@stsci.edu]>, Jonathan Eisenhamer <eisenhamer@stsci.edu [eisenhamer@stsci.edu|mailto:eisenhamer@stsci.edu] [eisenhamer@stsci.edu|mailto:eisenhamer@stsci.edu]> Subject:Re: FGS Parallel Observations
According to FGS instrument experts I’ve talked to over the years it’s possible to do science with the 1 FGS detector that’s not being used to do the guiding for any given observation. And the cal pipeline fully supports processing of FGS imaging mode science, for this reason. I believe most (or at least a lot of the) JWST instruments support pure parallel observations (e.g. NIRCam in parallel with NIRSpec, MIRI in parallel with NIRCam, etc.) and apparently in this case FGS in parallel with … something.
So the fundamental issue here is the fact that it appears that in the case of any instrument being operated in parallel, there’s no target id/name passed along from APT or PPS. I guess this kind of makes sense, due to the fact that there’s no way to know a priori what target will be showing up in the parallel field, so they leave it blank. But having the target id is a fundamental parameter to all of the association rules: the primary discriminator for associating exposures with one another is the fact that they have the same target id. So without a target id, we won’t get any level-3 ASN’s constructed for parallel observations from any instrument (including NIRCam, MIRI, etc.). Specific dither/mosaic patterns have been defined for certain instruments just for the purpose of creating useful science from the parallel instrument and hence there’s an expectation that those multiple parallel exposures will produce level-3 products. In order to do that, with the current ASN rules structures, we must have some kind of target id/name for the parallel exposures (even if it’s something simple like “parallel 1”). We just need something to match on.
-Howard
From:"Richard A. Shaw" <shaw@stsci.edu [shaw@stsci.edu|mailto:shaw@stsci.edu] [shaw@stsci.edu|mailto:shaw@stsci.edu]> Date:Thursday, May 14, 2020 at 8:25 AM To:David Davis <ddavis@stsci.edu [ddavis@stsci.edu|mailto:ddavis@stsci.edu] [ddavis@stsci.edu|mailto:ddavis@stsci.edu]> Cc:Howard Bushouse <bushouse@stsci.edu [bushouse@stsci.edu|mailto:bushouse@stsci.edu] [bushouse@stsci.edu|mailto:bushouse@stsci.edu]>, Mark Kyprianou <kyp@stsci.edu [kyp@stsci.edu|mailto:kyp@stsci.edu] [kyp@stsci.edu|mailto:kyp@stsci.edu]>, Jonathan Eisenhamer <eisenhamer@stsci.edu [eisenhamer@stsci.edu|mailto:eisenhamer@stsci.edu] [eisenhamer@stsci.edu|mailto:eisenhamer@stsci.edu]> Subject:Re: FGS Parallel Observations
Hi Dave,
I’m afraid I’m relatively clueless about FGS observations taken in parallel (as opposed to FGS operating as a guider). In a way, I am not surprised that no L-3 product is expected in the association generator rules. Per the comment on JDOX (Observatory Hardware->Fine Guidance Sensor):
“Unlike on HST, the Fine Guidance Sensors on JWST are expected to be used for guiding and calibration exclusively. Thus, at this time they are/not//available/for science proposals by general observers.”
I am not sure what is possible in APT for FGS: is using FGS in parallel an engineering mode? I also don’t know what a L-3 product would look like for FGS, nor do I know how such a product would be used to support guiding and calibration. That sounds like a question for Rossy or Matt Lallo.
I am not aware of any plan to change the target/Source ID upstream. As an aside, we did have trouble with CAOM picking up the waveband information for FGS, so to fix that issue we actually change the FILTER value from “NULL” to “OPEN” in CAOM.
Sorry I can’t be of more help.
Cheers,
Dick
From:David Davis <ddavis@stsci.edu [ddavis@stsci.edu|mailto:ddavis@stsci.edu] [ddavis@stsci.edu|mailto:ddavis@stsci.edu]> Date:Thursday, May 14, 2020 at 7:51 AM To:"Richard A. Shaw" <shaw@stsci.edu [shaw@stsci.edu|mailto:shaw@stsci.edu] [shaw@stsci.edu|mailto:shaw@stsci.edu]> Cc:Howard Bushouse <bushouse@stsci.edu [bushouse@stsci.edu|mailto:bushouse@stsci.edu] [bushouse@stsci.edu|mailto:bushouse@stsci.edu]>, Mark Kyprianou <kyp@stsci.edu [kyp@stsci.edu|mailto:kyp@stsci.edu] [kyp@stsci.edu|mailto:kyp@stsci.edu]>, Jonathan Eisenhamer <eisenhamer@stsci.edu [eisenhamer@stsci.edu|mailto:eisenhamer@stsci.edu] [eisenhamer@stsci.edu|mailto:eisenhamer@stsci.edu]> Subject:FGS Parallel Observations
Hi Dick,
I'm working on some parallel FGS observations (JP-1409) that need an
association for
level 3 processing. As you are aware the level 3 products are grouped
based on the
source or target ID to combine multiple observations. For the FGS
parallels these are all
set to NULL. Howard B. thinks that all parallel observations may have
the target ID set to
NULL.
Do you know if there is a document that covers how to group parallel
observations or
are there plans to change the target/source ID upstream so that the
association generator
can group parallel observations?
Thanks,
Dave
stscijgbot-jp
commented
4 years ago Comment by Pierre Chayer on JIRA:
After reading the previous notes and thinking about how to move forward with this issue, I realized that it would be easier for everyone if we, the FGS team, generated level-3 ASNs ourselves for these parallel observations. I believe that generating these level-3 ASNs is not too difficult to do. I hope I can get help if I ever have trouble generating them. I therefore propose to drop the jira ticket JP-1409.
stscijgbot-jp
commented
4 years ago
Issue JP-1409 was created on JIRA by Howard Bushouse:
Program 624 observation 8 consists of dithered NIRCam imaging exposures with FGS guider2 exposures obtained in parallel. The ASN generator creates image2 and image3 ASN's for the NIRCam exposures, but only creates image2 ASN's for the FGS guider2 exposures (no image3). The dithered FGS exposures should result in image3 ASN's, for processing in the calwebb_image3 pipeline.
The level-3 ASN rules might be too restrictive for FGS science exposures, in that the fact that FILTER and PUPIL values are always NULL for FGS exposures (no filters in the instrument).