Call to CALACS somehow invokes ancient ACSDTH step

[pllim]

UPDATE

Bugs to be fixed:

• [ ] Outdated message about PyDrizzle. This message should be updated.
• [ ] Why make me delete SPT files and then complains about the files not there?

Earlier log

@dborncamp is trying to process jcqtb4020_asn.fits but ran into an ancient and unexpected error message after ACS2D is completed:

<snipped>
*** ACS2D complete ***

CALACSBEG*** ACSDTH -- Version 9.2.0 (01-Jun-2017) ***
Begin    24-Aug-2017 14:22:24 EDT
The task PyDrizzle needs to be run in order to generate
a geometrically corrected, drizzle-combined product.
PyDrizzle requires PyRAF. See pyraf.stsci.edu for more details.
Warning    Output file `jcqtb4020_spt.fits' already exists.
ERROR:    CALACS processing NOT completed for jcqtb4020_asn.fits
ERROR:    CALACS processing NOT completed for jcqtb4020_asn.fits
ERROR:             status = 1021

ACSDTH is an ancient CALACS step that is no longer listed in ACS Data Handbook despite the existence of pkg/acs/calacs/calacs/acsdth.c.

To access the data:

ls /grp/hst/acs7/dborncamp/darktesting/ff/monthlong/jcqtb4*

To run:

calacs.e -v jcqtb4020_asn.fits

Content of ASN table:

<Table length=5>
   MEMNAME        MEMTYPE     MEMPRSNT
    str14          str14        bool  
-------------- -------------- --------
JCQTB4A5Q      EXP-DTH            True
JCQTB4BNQ      EXP-DTH            True
JCQTB4BWQ      EXP-DTH            True
JCQTB4DEQ      EXP-DTH            True
JCQTB4020      PROD-DTH           True

Trailer file with the failure is attached (because you need to delete the outputs before re-running, so this might get lost if not attached now):

jcqtb4020.tra.txt

[pllim]

So... The reason for the error message actually had nothing to do with PyDrizzle but because SPT files already exists! When I delete the SPT files, it complains about SPT files not found (???) but completes successfully:

*** ACS2D complete ***
Finished CCD processing...
CALACS: Building DTH products

CALACSBEG*** ACSDTH -- Version 9.2.0 (01-Jun-2017) ***
Begin    24-Aug-2017 15:50:20 EDT
The task PyDrizzle needs to be run in order to generate
a geometrically corrected, drizzle-combined product.
PyDrizzle requires PyRAF. See pyraf.stsci.edu for more details.
Warning    Can't find input file "jcqtb4a5q_spt.fits"
Warning    Can't find input file "jcqtb4bnq_spt.fits"
Warning    Can't find input file "jcqtb4bwq_spt.fits"
Warning    Can't find input file "jcqtb4deq_spt.fits"
End      24-Aug-2017 15:50:20 EDT

*** ACSDTH complete ***
Trying to open jcqtb4020_asn.fits...
Updated Global Header for jcqtb4020_asn.fits...
CALACS: Finished processing product 

End      24-Aug-2017 15:50:20 EDT

*** CALACS complete ***
CALACS completion for jcqtb4020_asn.fits

I think there are two things going on here:

• Outdated message about PyDrizzle. This message should be updated.
• Why make me delete SPT files and then complains about the files not there?

I am going to update the issue above with these two items that should be fixed.

[pllim]

@stsci-hack , do you have any comments on this reported behavior?

[dborncamp]

Also tried in an environment with IRAF but it crashed as well.

[mdlpstsci]

I cannot address why the ACSDTH step is not discussed in the ACS Data Handbook, but it is a current and viable processing step in the pipeline. The ACSDTH step creates the combined SPT file (dataset_spt.fits) which is comprised of the spt.fits files for each of the individual raw.fits files in the association table.

Scenario 1 In the current version of CALACS, if the combined SPT fits file already exists in the directory where the data is being processed (i.e., in the instance that all output products were obtained from the Archive, or the data has been previously processed in the local directory), CALACS has been designed not to overwrite the existing file, so the pipeline exits with an error. The error message could be clarified a bit.

Scenario 2 In the instance that the combined SPT file does not exist, but the individual SPT files corresponding to the RAW files exist, then processing completes successfully.

Scenario 3 In the instance that both the combined and individual SPT files do not exist, then the processing still completes successfully, but WARNING messages are issued stating the individual SPT files cannot be found. Strictly speaking, the SPT files are NOT needed for proper processing of data. Again, the WARNING messages could be clarified.

At the least, the WARNING and/or ERROR messages should be improved. One could argue that in Scenario 1, the processing could/should continue with the existing file being overwritten or copied to another name. You can imagine the copy/rename scheme could get out of hand if the same files are being processed repeatedly. The overwrite/copy/rename scheme would be more aligned with Scenario 3 (i.e., keep processing). Finally and based upon Pey-Lian's comment, it seems the PyDrizzle message should also be updated. If different behavior is desired for Scenario 1, then this issue will need to be discussed.

As for CALACS aborting under IRAF, can you just tell me more about this particular setup? I asked because it is not acceptable for a program to abort, rather than shutdown gracefully. Do you have a core dump? If so, can you do a simple "where" so I can see (hopefully) were the problem happened. No worries, if you do not have a dump. I would like to try to reproduce this situation to generate a core dump. ~

[dborncamp]

If this is a valid path, we should add it to our documentation...

@mdlpstsci Could you provide a small section to go into chapter 3 of the ACS Data Handbook, and make notes on where it should be added in other places in the confluence page: https://confluence.stsci.edu/display/HST/Chapter+3%3A+ACS+Calibration+Pipeline+-+DHBv9 ? We are currently trying to get out handbook updated so now is a good time to add it.

[pllim]

Yes, I think what Dave originally encountered was Scenario 1. 👍 for clarifying error message without overwriting existing SPT file.

As for including it in DHB, ACSDTH was in an older version of the handbook (e.g., see Figure 3.1 in http://documents.stsci.edu/hst/acs/documents/handbooks/DataHandbookv1/ACS_longdhb.pdf). So, we need to figure out why it was taken out in the first place, so we don't repeat the cycle. I wonder if Jennifer or Shireen remember the history.

[mdlpstsci]

ACSDTH is the optional last step of the CALACS pipeline that is activated by requested processing of an association table. If the filename to be processed by CALACS represents an association table file, this step was originally designed to perform the drizzle processing on the dithered images, hence the "DTH", in the association table to create a _dth.fits product. However, this step also creates a combined SPT file, as well as a combined trailer file which are comprised of data from the individual spt.fits or .tra files, respectively, for each of the individual _raw.fits files in the association table.

The _dth.fits files were empty products, presumably pending the drizzling algorithm. In 2002, creation of the _dth.fits product was eliminated as the drizzling algorithm is best done as a post-CALACS processing step. The creation of the combined SPT and trailer files, spt.fits and .tra, is the remaining functionality of this processing step.

I would like Warren @hack to comment on the above two paragraphs before any of the commentary is written in the handbook. Since the _dth.fits product is no longer generated as it did not generate a viable product, the remaining functionality is not so much a major processing step, as it is clean-up or close-up of the general CALACS processing.

[mdlpstsci]

Hi, Below are the final words with respect to the ACSDTH description. I do need to address the ERROR or WARNING messages you saw for clarity. Michele

ACSDTH is the optional last step of the CALACS pipeline that is activated by requested processing of an association table. If the filename to be processed by CALACS represents an association table file, this step was originally designed to perform the drizzle processing on the dithered images, hence the "DTH", in the association table to create a _dth.fits product. However, this step also creates a combined SPT file, as well as a combined trailer file which are comprised of data from the individual spt.fits or .tra files, respectively, for each of the individual _raw.fits files in the association table.

The CALACS-generated _dth.fits files were empty products pending the development of a pipeline-capable image combination algorithm such as drizzle. In 2002, routines invoked for the creation of the _dth.fits files were removed from CALACS in deference to the use of a separate post-CALACS drizzling algorithm. The creation of the combined _SPT and trailer files, for the association products spt.fits and .tra files, remains the sole functionality of this processing step.

[pllim]

Question for @dborncamp -- Do you think we still need to put this back into DHB? "ACSDTH is not really used to create DTH anymore but just SPT" can be confusing. But then again, such information should be documented somewhere.

[dborncamp]

If it is part of the pipeline, I would say yes as it is not documented anywhere else. But I will ask Norman.

[mdlpstsci]

I would like to look at the broader picture, and so I will ask are the combined SPT and trailer files useful and used for something now that the drizzling is done post-CALACS? Perhaps the archive is expecting these files and so there is some need for them to exist when there is an association? Not to malign these files, but if they are cruft now, then I suggest we eliminate them and all associated messages so as not to carry around ancient history just for the sake of carrying around ancient history.

[pllim]

SPT carries some technical metadata that do not necessarily appear in Primary Header of the products. I never used it much but did use it a few times during SM4 activities. Trailer file carries the full processing log.

Not sure if they are used in drizzling but @mcara should know.

[mcara]

To the best of my knowledge (stand-alone)drizzle needs neither _spt.fits nor _.tra files.

Also, maybe @stsci-hack or @philhodge may be able to provide additional perspective on the use of _spt.fits files.

[dborncamp]

I use the spt files sometimes. They have very useful information like the orbital position of the telescope, temperature at various locations, etc.

Unless @stsci-hack or @philhodge come up with some new information, Norman has decided that we should not include this in our new handbook/documentation instead re-do this function and at the very least change the warning to not reference IRAF routines and rename it to something to do with the spt files.

[mdlpstsci]

I will clarify a bit what I was trying to say earlier. The spt and .tra files exist for the individual images named in the association table. I was not suggesting any change to these individual files. I was only trying to determine if the combined spt and .tra files which are now the sole products of the ACSDTH processing step were actually used, and therefore critical, to another software application. Regardless of the answer to my query, I can imagine how the combined files are more convenient for the user.

I will update the WARNING message and determine the best way to handle the ERROR message (overwrite or rename an existing file). The pipeline should not fail in this very last step because there is an existing combined _spt.fits file.

[pllim]

@mdlpstsci , combined files can be useful when users only request the combined product (DRZ, DRC) from the Archive. Then, they can access SPT and TRA info without requested the individual files as well.

@mcara , I was under the impression that DrizzlePac updates the TRA file as it processes, is that not true? At least that's what the HSTCAL error message is suggesting.

[mcara]

@pllim I think you are right that drizzlepac may update TRA files if told so. Specifically, I think if drizzle step is run, then pipeline appends drizzle's log to .tra file. Nothing more... But please double-check me as I did not do much work on pipeline.

[mcara]

@pllim So, to re-formulate: astrodrizzle itself does not write to or need .tra - it is the pipeline that appends drizzlepac's log to .tra.

[pllim]

pipeline appends drizzle's log to .tra file

AFAIK, that pipeline is not HSTCAL, so must be some magical script on the DMS side. 🦄

[dborncamp]

The DMS regression tests do not have this issue for some reasion:

Begin    07-Sep-2017 02:42:12 UTC
2017250024212-I--------------- ACSDTH started ----------------------------------
The task PyDrizzle needs to be run in order to generate
a geometrically corrected, drizzle-combined product.
PyDrizzle requires PyRAF. See pyraf.stsci.edu for more details.
Trying to open jcqtb4a5q_spt.fits...
Read in Primary header from jcqtb4a5q_spt.fits...
Created output SPT rootname jcqtb4020_spt.fits...

Updated output SPT file to reflect 4 extensions...

Trying to open jcqtb4bnq_spt.fits...
Read in Primary header from jcqtb4bnq_spt.fits...
Trying to open jcqtb4bwq_spt.fits...
Read in Primary header from jcqtb4bwq_spt.fits...
Trying to open jcqtb4deq_spt.fits...
Read in Primary header from jcqtb4deq_spt.fits...
End      07-Sep-2017 02:42:12 UTC

*** ACSDTH complete ***
Trying to open jcqtb4020_asn.fits...
Updated Global Header for jcqtb4020_asn.fits...
CALACS: Finished processing product

End      07-Sep-2017 02:42:12 UTC

*** CALACS complete ***```

[pllim]

Probably because they started out with individual SPT files but not the ASN one.

[mdlpstsci]

HI,

This is exactly the reason. As long as the combined ASN SPT does not exist, and the individual RAW SPT images do exist, there will be no warning or error messages. M

From: "P. L. Lim" notifications@github.com Reply-To: spacetelescope/hstcal reply@reply.github.com Date: Monday, September 11, 2017 at 9:48 AM To: spacetelescope/hstcal hstcal@noreply.github.com Cc: Michele De La Pena mdelapena@stsci.edu, Mention mention@noreply.github.com Subject: Re: [spacetelescope/hstcal] Call to CALACS somehow invokes ancient ACSDTH step (#199)

Probably because they started out with individual SPT files but not the ASN one.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/spacetelescope/hstcal/issues/199#issuecomment-328534460, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AcsYntabfqU1hBGw1hBYzeTEHgcKtfbBks5shTopgaJpZM4PB02M.

[mdlpstsci]

I have responded in explanation or with changes to code to the comments made on the PR#286 which corresponds to this issue. Please review again.

[jamienoss]

Reopening due to failed regression test c.f. issue #290

[mdlpstsci]

This issue is fundamentally just about clarification of CALACS messages to avoid misunderstanding. The actual bug found during examination of this issue will be submitted as a separate Git Issue.