Review B7.5 release candidate test results from DMS

stscijgbot-jp commented 4 years ago

Issue JP-1445 was created on JIRA by Howard Bushouse:

Review results of DMS test runs of the initial B7.5 deliveries from all sub-systems on the D-string side 2. Test run information is cataloged at https://innerspace.stsci.edu/pages/viewpage.action?pageId=209800770]. Review results for problems related directly to the calibration pipeline and associations software and file new JP tickets as appropriate.

stscijgbot-jp commented 4 years ago

Comment by Howard Bushouse on JIRA:

complete_short_04: 11-May-2020 ████████████████████████████████████████████████████

1108-UnsupportedOperandTypes Known error due to APERNAME='UNKNOWN' in some exposures Not a Cal problem
1128-AssignWcsException ** Known error due to on-going updates for NIRSpec lamp processing
1129-UnexpectedSlitYRangeArgument ** Known error due to on-going updates for NIRSpec lamp processing
1177-KeyError calwebb_detector1 error in rscd correction step for MIRI FASTGRPAVG exposures JP-1446 has been filed to investigate and fix
1178-ValueError calwebb_image3 error in source_catalog step for some MIRI and NIRCam exposures JP-1440 has been filed to investigate and fix calwebb_image3 error in source_catalog step information added to JP-1440

stscijgbot-jp commented 4 years ago

Comment by Howard Bushouse on JIRA:

complete_medium_03: 11-May-2020 ███████████████████████████████████████████████████

1108-UnsupportedOperandTypes ** Same as above. Not a Cal problem.
1143-Shapes Same calwebb_detector1 error as seen before for NIRCam exposures that are full frame, but claim to be subarray. Not a Cal problem
1144-BadRefFileSliceIndexes calwebb_detector1 error due to long-known problem with NRS2 detector subarray corner values in readnoise reference file Not a Cal problem
1177-KeyError calwebb_detector1 error in rscd correction step for MIRI FASTGRPAVG exposures, as reported above JP-1446 has been filed
1182-KeyError Same as 1177 above (calwebb_detector1 rscd step error) for a MIRI dark exposure JP-1446 has been filed
1178-ValueError Same calwebb_image3 error in source_catalog reported above JP-1440 in progress to fix
1180-OutBoundsIndex calwebb_image3 error in source_catalog step for a NIRISS imaging observation Added to JP-1440

stscijgbot-jp commented 4 years ago

Comment by Howard Bushouse on JIRA:

complete_long_05: 11-May-2020 ███████████████████████████████████████████████████

1108-UnsupportedOperandTypes ** Same as above. Not a Cal problem.
1129-UnexpectedSlitYRangeArgument ** Known error due to on-going updates for NIRSpec lamp processing
1137-Shapes calwebb_detector1 error in dark_current step for NIS_SOSS exposure that says SUBARRAY='FULL', but image is 256x2048 Problem in level-1b
1143-Shapes Same calwebb_detector1 error as seen before for NIRCam exposures that are full frame, but claim to be subarray. Not a Cal problem
1172-ValueError calwebb_spec2 error in resample_spec step for MIRI LRS slitless exposure that's incorrectly labeled with EXP_TYPE='MIR_LRS-FIXEDSLIT' Shouldn't be going through resample_spec step ** Error in meta data upstream
1177-KeyError Same as above for MIRI RSCD step error JP-1446 has been filed
unknown->=1ProductsFailed calwebb_spec2 error in bkg_subtract step for NIS_WFSS exposures Missing columns in input source_cat (cat.ecsv) file ** JP-1447 has been filed to investigate
1178-ValueError Same calwebb_image3 error in source_catalog reported above JP-1440 in progress to fix
1180-OutBoundsIndex Same calwebb_image3 error in source_catalog step for a NIRISS imaging observation reported above Added to JP-1440
1187-ErrDeterminingBestRef Error in calwebb_image3 determining APCORR ref file for MIR_TACQ exposures There are actually two problems here: The calwebb_image3 pipeline is mistakenly loading all input ASN members, including targ_acq's. That's wrong and needs to be fixed. JP-1448 has been filed. The MIRI APCORR rmap in CRDS doesn't have an entry for MIR_TACQ; it should. CRDS-350 has been filed.
1186-ValueError calwebb_spec3 error in mrs_imatch step classic "output array is read-only" error when calling numpy functions involved in sigma clipping ** Filed JP-1450 to investigate

stscijgbot-jp commented 4 years ago

Comment by Howard Bushouse on JIRA:

mt1-OTB: 13-May-2020 ███████████████████████████████████████████████████

1159-ErrDeterminingBestRef calwebb_image2 error determining distortion and flat ref files; more than one match clause NRC_IMAGE, NRCALONG, F444W, PUPIL='N/A'
1190-KeyError calwebb_spec2 error in background subtraction step for NIRISS WFSS data; missing columns in source catalog same as reported above Fixed in JP-1447**
1178-ValueError same calwebb_image3 error in source_catalog as above Fixed in JP-1440
1186-ValueError same calwebb_spec3 error in mrs_imatch as above Fixed in JP-1450
unknown-AssignmentReadOnly calwebb_image3 error in source_catalog step when processing MIRI BRIGHTSKY subarray "No matching row found in APCORR ref file" JP-1454 filed to investigate**

stscijgbot-jp commented 4 years ago

Comment by Howard Bushouse on JIRA:

nOps6: 13-May-2020 ███████████████████████████████████████████████████

1190-KeyError same calwebb_spec2 background subtraction step problem with source catalog as above Fixed in JP-1447
1178-ValueError same calwebb_image3 error in source_catalog as above Fixed in JP-1440
1186-ValueError same calwebb_spec3 error in mrs_imatch as above Fixed in JP-1450
unknown-AssignmentReadOnly same calwebb_image3 source_catalog error for MIRI as above JP-1454 filed to investigate

stscijgbot-jp commented 4 years ago

Comment by Howard Bushouse on JIRA:

nOps7: 13-May-2020 ███████████████████████████████████████████████████

No reported errors.

stscijgbot-jp commented 4 years ago

Comment by Howard Bushouse on JIRA:

GSEG3DR4: 13-May-2020 ███████████████████████████████████████████████████

No LEVEL_2 or LEVEL_3 errors.

stscijgbot-jp commented 4 years ago

Comment by Howard Bushouse on JIRA:

complete_short_04: 14-May-2020 ███████████████████████████████████████████████████

1108-UnsupportedOperandTypes calwebb_image2 error in resample step for NIRCam SUB400P image due to APERNAME='UNKNOWN' upstream issue
1177-KeyError calwebb_detector1 error in rscd correction step for MIRI FASTGRPAVG exposures Fixed in JP-1446
1188-UnexpectedSlitYRangeArgument ** Known error due to on-going updates for NIRSpec lamp processing
1191-ValueError calwebb_nrslamp-spec2 error in assign_wcs due to no ref files for lamp exposures Known error due to on-going updates for NIRSpec lamp processing
1178-ValueError calwebb_image3 errors in source_catalog step as seen before Fixed in JP-1440
1180-OutBoundsIndex calwebb_image3 error in source_catalog step Added JP-1454

stscijgbot-jp commented 4 years ago

Comment by Howard Bushouse on JIRA:

complete_medium_03: 14-May-2020 ███████████████████████████████████████████████████

1108-UnsupportedOperandTypes calwebb_image2 error due to APERNAME='UNKNOWN' SDP problem
1143-Shapes calwebb_detector1 error with NIRCam subarray exposures labeled as full-frame SDP or PPS problem
1444-BadRefFileSliceIndexes calwebb_detector1 error due to long-known problem with NRS2 detector subarray corner values in readnoise ref file Reference file problem
1177-KeyError calwebb_detector1 error in RSCD step for MIRI FASTGRPAVG exposures Fixed in JP-1446
1182-KeyError calwebb_dark error in RSCD step for MIRI FASTGRPAVG exposures Fixed in JP-1446
1178-ValueError calwebb_image3 error in source_catalog step Fixed in JP-1440
1180-OutBoundsIndex calwebb_image3 error in source_catalog step Fixed in JP-1454

stscijgbot-jp commented 4 years ago

Comment by Howard Bushouse on JIRA:

EvaluationSuite05: 13-15-May-2020 ███████████████████████████████████████████████████

1116-ErrDeterminingBestRef calwebb_wfs-image2 pipeline error finding distortion and flat ref files (more than one match) CRDS problem
1143-Shapes calwebb_detector1 error with NIRCam exposures claiming to be SUB320, but data are actually full-frame Upstream problem
1177-KeyError calwebb_detector1 RSCD step error for MIRI image with FASTGRPAVG Fixed by JP-1446
1188-UnexpectedSlitYRangeArgument calwebb_spec2 error with NRS internal lamp exposures Known problem related to on-going internal lamp processing development
1190-KeyError calwebb_spec2 error in background step due to new source catalog column names Fixed by JP-1447
1191-ValueError calwebb_spec2 error with NRS internal lamp exposures Known problem related to on-going internal lamp processing development
1192-NoDataOnDetectorError calwebb_spec2 error because no spectra on NRS2 detector Expected error for some NIRSpec filter+grating+slit combinations
1140-AttributeError calwebb_image3 error in skymatch step for NIRCam observation reports no sources found and zero sky background, so suspect the error is just due to garbage data ** Not worth pursuing at this point
1171-ValueError calwebb_tso3 error in extract_1d step for NIRISS SOSS JP-1462 has been filed
1178-ValueError calwebb_image3 error in source_catalog Fixed in JP-1440
1186-ValueError calwebb_spec3 error in mrs_imatch step for MIRI MRS data Fixed in JP-1450
1187-ErrDeterminingBestRef calwebb_image3 error selecting apcorr ref file for MIR_TACQ CRDS problem
unknown-IndexOutOfRangeA-WfsCombineStepError calwebb_wfs-image3 error for NIRCam WFSC exposure pairs Problem is in OSF and SDP clobbering data from multiple processing runs. Not a CAL problem.
unknown-NoSuchFile-WfsCombineStepError calwebb_wfs-image3 error with output file not being found Assuming problem is again due to data getting clobbered by multiple runs (as above)
unknown-OSError-WfsCombineStepError calwebb_wfs-image3 error Assuming problem is again due to data getting clobbered by multiple runs (as above)
unknown-ValueError calwebb_wfs-image3 error same as above
unknown-Image3PipelineError calwebb_image3 error in source_catalog step for MIRI images no matching row found in apcorr ref file Fixed in JP-1454**
unknown-ErrorExit calwebb_spec3 error in cube_build step for NIRSpec IFU data appears to be a Memory error? No explicit error/exception message from the cal pipeline

stscijgbot-jp commented 4 years ago

Comment by Howard Bushouse on JIRA:

long_running_02: May 15-18, 2020 ███████████████████████████████████████████████████

1130-MemoryError calwebb_tso1 memory error processing NIRCam exposure some reference the inability to allocate a 16.9GB array, which is perhaps understandable, while others reference a 3.5GB array All occurrences are for jw00721009001_03103_00001-seg00x_nrcb1, for segments 1, 4, 5, and 7, implying that segments 2, 3, and 6 succeed For segments 1, 4, and 5 the error occurs in the jump step (twopoint_difference.py line #⁠45) when attempting to create "saturated_pixels" array ** For segment 7 it occurs already in the dq_init step when trying to do "output_model=input_model.copy()"
1188-UnexpectedSlitYRangeArgument ** calwebb_spec2 error related to on-going NIRSpec internal lamp processing development
1189-ValueError calwebb_spec2 error in resample_spec when processing MIRI slitlessprism exposure erroneously labeled as fixed-slit. Not a Cal problem
1193-MemoryError ** Appears to be same errors as 1130-MemoryError. Not sure why they're in 2 error logs.
226-CalMissingInputFile ** calwebb_spec2 error due to missing input file
1106-MemoryError calwebb_spec3 error processing large NIRSpec IFU ASN automatic retry after initial memory error worked and resulted in successful completion of the pipeline!

stscijgbot-jp commented 4 years ago

Comment by Howard Bushouse on JIRA:

wfr4_sir2.5: May 17, 2020 ███████████████████████████████████████████████████

No error lists found

stscijgbot-jp commented 4 years ago

Comment by Howard Bushouse on JIRA:

GSEG3DR4: May 18, 2020 ███████████████████████████████████████████████████

No failures

stscijgbot-jp commented 4 years ago

Comment by Howard Bushouse on JIRA:

complete_long_05: May 16, 2020 ███████████████████████████████████████████████████

1171-ValueError calwebb_tso3 error in extract_1d step when processing NIRISS SOSS observations using subarray SUBSTRIP256 JP-1462 has been filed to investigate
unknown-AssignmentReadOnly-CalibrationWrapperFailedError-IndexOutOfRangeA-WfsCombineStepError wfs_combine error JP-1463 has been filed to investigate * UPDATE: This is due to issues with OSF and SDP. Not a CAL problem. Ignoring.*

No other new or previously unknown errors.

stscijgbot-jp commented 4 years ago

Comment by Howard Bushouse on JIRA:

SOC43: May 17, 2020 ███████████████████████████████████████████████████

No level-2 or level-3 errors.

stscijgbot-jp commented 4 years ago

Comment by Howard Bushouse on JIRA:

sk21: May 17. 2020 ███████████████████████████████████████████████████

No error reports.

stscijgbot-jp commented 4 years ago

Comment by Howard Bushouse on JIRA:

nOPS7: May 18, 2020 ███████████████████████████████████████████████████

No error reports (yet)

stscijgbot-jp commented 4 years ago

Comment by David Davis on JIRA:

nOPS7: May 20, 2020 ███████████████████████████████████████████████████

1116-ErrDeterminingBestRef calwebb_wfs-image2 pipeline error finding distortion and flat ref files (more than one match) CRDS problem

stscijgbot-jp commented 4 years ago

Comment by David Davis on JIRA:

EvaluationSuite05: May 21, 2020 ███████████████████████████████████████████████████

1116-ErrDeterminingBestRef calwebb_wfs-image2 error determining distortion and flat ref files CRDS ref file issue
1143-Shapes ** calwebb_detector1 error on NIRCam images with incorrect SUBARRAY name
1188-UnexpectedSlitYRangeArgument ** known calwebb_nrslamp-spec2 issue with NRS internal lamp exposures
1191-ValueError ** known calwebb_nrslamp-spec2 issue with NRS internal lamp exposures
1140-AttributeError calwebb_image3 error skymatch step reports no sources found and zero sky background, so suspect the error is just due to garbage data ** Not worth pursuing at this point
1171-ValueError calwebb_tso3 error in extract_1d step Fixed in JP-1462 (not included in 0.16.1)
1187-ErrDeterminingBestRef calwebb_image3 error determining apcorr ref file for MIR_TACQ See CRDS-350 ** Fixed in CRDS_CONTEXT 0617

stscijgbot-jp commented 4 years ago

Comment by David Davis on JIRA:

nOPS7: May 21, 2020 ███████████████████████████████████████████████████

1116-ErrDeterminingBestRef calwebb_wfs-image2 pipeline error finding distortion and flat ref files (more than one match) CRDS problem

stscijgbot-jp commented 4 years ago

Comment by David Davis on JIRA:

MinAsn-11 May 21, 2020 ███████████████████████████████████████████████████

clean run!

stscijgbot-jp commented 4 years ago

Comment by David Davis on JIRA:

nOPS6: May 21, 2020 ███████████████████████████████████████████████████

1212-OutBoundsIndex calwebb_spec3 error in combine_1d step for NIRISS WFSS data "IndexError: index 1 is out of bounds for axis 0 with size 1" Filed JP-1473**

stscijgbot-jp commented 4 years ago

Comment by David Davis on JIRA:

sk21d: May 22, 2020 ███████████████████████████████████████████████████

1122-MemoryErrorExit88 calwebbspec3 (jw00651-o003{source_id}_nircam) Needs more investigation (file ticket)
1130-MemoryError (same as May 18 above?) Detector1Pipeline (jw00649-o003) MemoryError: Unable to allocate 9.38 GiB for an array with shape (200, 3, 2048, 2048) and data type float32
1178-ValueError calwebb_image3 error in source_catalog positions must not contain any non-finite (e.g. NaN or inf) Appears to have a different error signature than the 3 that were fixed in JP-1440**
unknown-25675503 calwebb_guider error Computed reference file slice indexes are incompatible with size of reference data array CRDS?

stscijgbot-jp commented 4 years ago

Comment by David Davis on JIRA:

10006: May 22, 2020 ███████████████████████████████████████████████████

A clean run

stscijgbot-jp commented 4 years ago

Comment by David Davis on JIRA:

long_running_02: May 23, 2020 ███████████████████████████████████████████████████

1130-MemoryError, same as May 18 2020 long running error
1188-UnexpectedSlitYRangeArgument calwebb_spec2 error related to on-going NIRSpec internal lamp processing development Same as seen in May 15-18 long running above,
1189-ValueError ** same as May 15-18 error, not a calibration issue
1193-MemoryError, another segmented file error, related to 1130?
1234-MemoryError, error in get_calib_recipe.py not a calibration issue
1106-MemoryError calwebb_spec3 error processing NIRSpec IFU data pipeline completes successfully (saves s3d and x1d products), but the log records a memory error that was thrown on a previous processing attempt

stscijgbot-jp commented 4 years ago

Comment by David Davis on JIRA:

complete_long_05: May 23, 2020 ███████████████████████████████████████████████████

1108-UnsupportedOperandTypes ** Same as above. Not a Cal issue.
1116-ErrDeterminingBestRef ** CRDS issue
1137-Shapes ** Incorrect subarray name (See May 11 above)
1143-Shapes ** Not a cal issue, see May 13-15 above
1171-ValueError Ticket JP-1462**
1187-ErrDeterminingBestRef ** CRDS issue
1188-UnexpectedSlitYRangeArgument ** same as above
1189-ValueError ** same as above
1191-ValueError ** known error (See May 13-15 run)
1202-TypeError ** It looks like all images were rejected so when looking for the min value for the sky value a none is returned.
1204-ErrDeterminingBestRef ** CRDS issue
1233-NoKywdValueSet ** fitsbuilder, needs keyword set, Not a cal issue

spacetelescope / jwst

Review B7.5 release candidate test results from DMS #4924