stscijgbot-hstdp
commented
1 year ago Comment by Michele De La Pena on JIRA:
Rick White I took what I thought would be a quick look at this ticket because it sounded more like (possibly) an operations issue than something I should handle. Please note I am looking at the data from dlhlalab4 in the subdirectories of /ifs/archive/ops/hst/public/. I also understand these were just a few examples of what is thought to be a larger issue. Here is what I found for the datasets specified.
ien6/hst_16683_03
This does not seem to be the dataset described in Rick's ticket???
Having said this...not sure what is going on here. This dataset has all the disk files with a date of 25/26 April 2023. However, the IR/f125w FLTs all have a DATE keyword of '2022-06-14' and the UVIS/f555w/f814w files all have a DATE keyword of '2023-04-21'. The inconsistency is between the IR and UVIS processing of data for this visit. This sounds like an operations issue, but Rick should also check the dataset name.
jec4/hst_16231_06
- The disk dates of the files do range from 14 April 2022 to 14 June 2022. However, the SVM FLTs all have DATE = '2022-04-13', and the trailer files all have an internally documented "AstroDrizzle Version ... started at:" date of 14 April 2022, though it is written in confusing style, grumble, of (14/04/2022). I do see all of the files associated with f165lp were written to disk in June although they were run through AstroDrizzle in April! All of these issues are outside of my bailiwick and someone else in team-violet can take a look at this issue. However, I also noticed the four f150lp science images are completely empty/nada/nill. All the pixels in the RAW SCI extensions of the four files are zero. While the software did report
"Input data is constant. No local peaks can be found."
the data continued to be processed. Indeed, the source catalogs have no found sources. This is an issue for me to handle - at the least with more explicit messages.
FYI: The problem of the SVM DR? files not having a DATE keyword in the PHDU has been fixed for the next version of Drizzlepac.
ie9t/hst_16274_04
All of these files have disk dates of 25 Apr (2023). The FLTs have DATE keyword values of DATE = '2023-04-21' and the trailer files have an "AstroDrizzle started" at date of 25 Apr 2023. It looks like this dataset is now fine.
Issue HLA-987 was created on JIRA by Rick White:
There is a problem in the HST public cache for HAP products: a single visit sometimes has files from multiple versions of data processing. This can be identified from a very large range of file creation times in the cache directory. Here are some examples:
|| visit || instrument || file time range || | ien6/hst_16683_03 | wfc3/uvis | 151.7 days| | jec4/hst_16231_06 | acs/sbc | 61.8 days | | ie9t/hst_16274_04 | wfc3/uvis | 73.5 days |
The
file time rangecolumn gives the time elapsed between the creation of the oldest file in the directory and the newest file (for that instrument). Most visits have creation time ranges from a few minutes to at most a few hours for all the files in the cache directory. A time range of 60 days or more (like these) certainly indicates that the visit was processed and then later reprocessed, but some of the files did not get updated during the reprocessing.The first visit
ien6/hst_16683_03is a typical example. It has 26 wfc3/uvis files that were created on 2022 Jul 16 and 32 wfc3/uvis files created on 2022 Dec 14. The old files are all for F814W, while the new files include both wfc3/uvis F555W and the total images and catalogs.There are two questions:
Why were some of the products not reprocessed when the visit directory was updated?
Why were the old (unreplaced) files allowed to remain on disk in the directory rather than being cleaned out before the new products were copied into the cache?
Currently the mix of old and new data files are in the CAOM database and are available for download (despite not being processed at the same time). That means there are some serious problems with the data products. For example, for visit
ien6/hst_16683_03, the f814w catalog files were generated in the old processing, while the f555w and total catalog files are from the new processing. Those catalogs are surely not consistent with each other (and neither are the images, which are supposed to be aligned).