search

spacetelescope / drizzlepac

AstroDrizzle for HST images.
https://drizzlepac.readthedocs.io
BSD 3-Clause "New" or "Revised" License
50 stars 39 forks source link

SVM: Master ticket itemizing the bugs found in 1K test run done 09 March 2021 #960

Closed stscijgbot-hstdp closed 2 years ago

stscijgbot-hstdp commented 3 years ago

Issue HLA-498 was created on JIRA by Michele De La Pena:

The issues dealing with S_REGION and Grism/Prism processing are high priority.  The catalog fixes and associated quality control updates should be considered lower priority.

(1) ia0m04/HAP_ia0m04.out:IndexError: index 0 is out of bounds for axis 0 with size 0 ==> This is a bug in computing the S_REGION which Warren and I have already discussed. Warren is working this issue. (HLA-500)

Cascading from (1) is: (2) ia0m04/HAP_ia0m04.out: raise KeyError(f"Keyword {keyword!r} not found.") ia0m04/HAP_ia0m04.out:KeyError: "Keyword 'PHOTFLAM' not found." ==> I will write a little utility to ease the burden of knowing if the keyword lives in the Primary or in the Science extension. Michele is working this issue. (HLA-502)

(3) ia1s70/HAP_ia1s70.out:KeyError: 'RA' ==> This is in the svm_quality_analysis.py.

This error stems from the source list cross-matching to GAIA returning with 0 cross-matches, and yet the code does not know, so it tries to read the file with the cross-match information. All this occurs in the svm_quality_analysis code, not in the operational code. [Warren has confirmed that this can be processed successfully with all PRs merged.]

(4) ia1s70/HAP_ia1s70.out:TypeError: init() missing 1 required positional argument: 'log_level' ==> This is in the svm_quality_analysis.py and is an incorrect call to the constructor. def init(self, filename, num_images_mask, log_level)

Warren has confirmed that this can be processed successfully with all PRs merged.

(5) iabk05/HAP_iabk05.out: raise ValueError iabk05/HAP_iabk05.out:ValueError ==> Indicates self.selected_fit = None (FAILED previous)

This can occur for any number of reasons during the attempt to align the images to GAIA. However, this exception is already handled appropriately by the code.

(6) ib1319/HAP_ib1319.out: raise ValueError('fwhm must be positive.') ib1319/HAP_ib1319.out:ValueError: fwhm must be positive. ==> catalog_utils.py - Point Catalog. From the decutils.find_point_sources(), the returned source_fwhm is negative. This generates an exception on decutils.UserStarFinder().

Warren will address this. HLA-501

(7) ib2p57/HAP_ib2p57.out:FileNotFoundError: [Errno 2] No such file or directory: 'hst_11613_57_wfc3_uvis_f606w_ib2p57_point-cat.ecsv' ==> This error is in svm_quality_analysis.py. In this case the Point catalog failed the CR threshold, so both catalogs were rejected. However, Quality Assurance is now going to run, and there are no catalogs to evaluate.

CR threshold computation revised in Drizzlepac PR #⁠972 which should restore these catalogs as outputs.

(8) ibbso1/HAP_ibbso1.out:AttributeError: 'NoneType' object has no attribute 'write' ==> In catalog_utils.py. Identify sources (Point) daofind returns a None, and there is no trap for this.

Warren addressed this in Drizzlepac PR #⁠965 through HLA-501.

(9) ibcd61/HAP_ibcd61.out:RuntimeError: Number of output coordinates exceeded allocation (2) ==> In align_utils.py - match_default_fit(). Is this the case where multiple sources are being matched?

Warren will confirm whether latest PRs will address this issue or not.

(10) ibsf01/HAP_ibsf01.out:numpy.core._exceptions._ArrayMemoryError: Unable to allocate 494. MiB for an array with shape (8074, 8014) and data type float64 ==> BIG array.

(11) ibtq25/HAP_ibtq25.out:ValueError: need at least one array to concatenate ==> Similar to (6). Both catalogs rejected due to CR threshold. Need to update svm_quality_analysis.py tests.

Rejection due to CR threshold may be reasonable. All but 2 of the exposures are single exposures in a unique filter resulting in a total detection image dominated by un-flagged CRs. The 2 IR exposures include 1 Grism exposure and a single F160W exposure with no drizzling done. We will need to add logic to protect against trying to work with non-existent catalogs.

(12) ibtq25/HAP_ibtq25.out:UnboundLocalError: local variable 'diagnostic_obj' referenced before assignment ==> In svm_quality_analysis.py. No catalogs. Need to update svm_quality_analysis.py.

This dataset triggers a number of error conditions in the svm_quality_analysis code that were not handled well enough by the current logic. Warren is investigating.

(13) id8453/HAP_id8453.out:TypeError: 'NoneType' object is not subscriptable ==> In svm_quality_analysis.py. Error in trying to create GAIA catalog. Need to update svm_quality_analysis.py.

(14) j8dnd5/HAP_j8dnd5.out:TypeError: unsupported operand type(s) for |: 'int' and 'NoneType' ==> align_utils.py Looks like there is no crmap which is an argument to np.bitwise_or(). No protection for the variable being None.

Warren will address this in Drizzlepac PR #⁠965 under HLA-501.

(15) j8dnd5/HAP_j8dnd5.out: raise KeyError(f"Keyword {keyword!r} not found.") j8dnd5/HAP_j8dnd5.out:KeyError: "Keyword 'HDRNAME' not found." ==> When handling the Grism/Prism data when resetting the active WCS from a headerlet, the HDRNAME is used. This keyword is not available. (HLA-499)

(16) j8ep07/HAP_j8ep07.out:ValueError: not enough values to unpack (expected 3, got 0) ==> In svm_quality_analysis.py. Trying to work with an empty *.ecsv file. While the svm_quality_analysis tests need to be rewritten to defend against missing data, how did an empty point catalog even get written? i2021068170115 INFO src=drizzlepac.haputils.catalog_utils- Wrote catalog file 'hst_9565_07_acs_sbc_f150lp_j8ep07_point-cat.ecsv' containing 0 sources 2021068170115 INFO src=drizzlepac.haputils.catalog_utils- Wrote catalog file 'hst_9565_07_acs_sbc_f150lp_j8ep07_segment-cat.ecsv' containing 9 sources 2021068170115 INFO src=drizzlepac.haputils.catalog_utils- Wrote catalog file 'hst_9565_07_acs_sbc_f165lp_j8ep07_point-cat.ecsv' containing 0 sources 2021068170115 INFO src=drizzlepac.haputils.catalog_utils- Wrote catalog file 'hst_9565_07_acs_sbc_f165lp_j8ep07_segment-cat.ecsv' containing 9 sources

Michele is working this issues as HLA-504.

(17) j8mb9b/HAP_j8mb9b.out:RuntimeError: Number of output coordinates exceeded allocation (1000) ==> We have seen this message previously. Working with tweakwcs/matchutils and calling stimage.

(18) j97e06/HAP_j97e06.out:IndexError: list index out of range ==> This is in the handling of Grism/Prism data. Trying to restore WCS from headerlet, and there are two headerlets with same HDRNAME. Cascades to another exception IndexError. (HLA-499)

Warren confirmed this dataset can be processed successfully with all PRs merged.

(19) Follow on to (16) Fix computation of total exposure time which is used in the cosmic ray threshold catalog rejection algorithm.

==> Warren has addressed this Git PR#⁠972.

 (20) The "reject catalogs" and "diagnostic mode" control whether or not the .ecsv and/or .reg catalogs files are written.  Clean up the logic in hapsequencer and catalog_utils so this works properly. HLA-505.

 

stsci-hack commented 2 years ago

Closed by #972, #965, and #962 (at the very least).