Runtime problem

[rgriffiths1947]

(kcwidrp) rgrifft1@GriffsMacBook sci % ls KB.20210609.02927.fits KB.20210609.29417.jpg KB.20210609.02927.fits.gz KB.20210609.30710.fits KB.20210609.02927.jpg KB.20210609.30710.fits.gz KB.20210609.04049.fits KB.20210609.30710.jpg KB.20210609.04049.fits.gz KB.20210609.31980.fits KB.20210609.04049.jpg KB.20210609.31980.fits.gz KB.20210609.04077.fits KB.20210609.31980.jpg KB.20210609.04077.fits.gz KB.20210609.33367.fits KB.20210609.04077.jpg KB.20210609.33367.fits.gz KB.20210609.04104.fits KB.20210609.33367.jpg KB.20210609.04104.fits.gz KB.20210609.34632.fits KB.20210609.04104.jpg KB.20210609.34632.fits.gz KB.20210609.04131.fits KB.20210609.34632.jpg KB.20210609.04131.fits.gz KB.20210609.35897.fits KB.20210609.04131.jpg KB.20210609.35897.fits.gz KB.20210609.04158.fits KB.20210609.35897.jpg KB.20210609.04158.fits.gz KB.20210609.37311.fits KB.20210609.04158.jpg KB.20210609.37311.fits.gz KB.20210609.04185.fits KB.20210609.37311.jpg KB.20210609.04185.fits.gz KB.20210609.38583.fits KB.20210609.04185.jpg KB.20210609.38583.fits.gz KB.20210609.04212.fits KB.20210609.38583.jpg KB.20210609.04212.fits.gz KB.20210609.39847.fits KB.20210609.04212.jpg KB.20210609.39847.fits.gz KB.20210609.04240.fits KB.20210609.39847.jpg KB.20210609.04240.fits.gz KB.20210609.41370.fits KB.20210609.04240.jpg KB.20210609.41370.fits.gz KB.20210609.04266.fits KB.20210609.41370.jpg KB.20210609.04266.fits.gz KB.20210609.42640.fits KB.20210609.04266.jpg KB.20210609.42640.fits.gz KB.20210609.04810.fits KB.20210609.42640.jpg KB.20210609.04810.fits.gz KB.20210609.43920.fits KB.20210609.04810.jpg KB.20210609.43920.fits.gz KB.20210609.04985.fits KB.20210609.43920.jpg KB.20210609.04985.fits.gz KB.20210609.45264.fits KB.20210609.04985.jpg KB.20210609.45264.fits.gz KB.20210609.05161.fits KB.20210609.45264.jpg KB.20210609.05161.fits.gz KB.20210609.46525.fits KB.20210609.05161.jpg KB.20210609.46525.fits.gz KB.20210609.05337.fits KB.20210609.46525.jpg KB.20210609.05337.fits.gz KB.20210609.48201.fits KB.20210609.05337.jpg KB.20210609.48201.fits.gz KB.20210609.05513.fits KB.20210609.48201.jpg KB.20210609.05513.fits.gz KB.20210609.49464.fits KB.20210609.05513.jpg KB.20210609.49464.fits.gz KB.20210609.05688.fits KB.20210609.49464.jpg KB.20210609.05688.fits.gz KB.20210609.50723.fits KB.20210609.05688.jpg KB.20210609.50723.fits.gz KB.20210609.05864.fits KB.20210609.50723.jpg KB.20210609.05864.fits.gz KB.20210609.52372.fits KB.20210609.05864.jpg KB.20210609.52372.fits.gz KB.20210609.06079.fits KB.20210609.52372.jpg KB.20210609.06079.fits.gz KB.20210609.53037.fits KB.20210609.06079.jpg KB.20210609.53037.fits.gz KB.20210609.06178.fits KB.20210609.53037.jpg KB.20210609.06178.fits.gz KB.20210609.53903.fits KB.20210609.06178.jpg KB.20210609.53903.fits.gz KB.20210609.06301.fits KB.20210609.53903.jpg KB.20210609.06301.fits.gz KB.20210609.53963.fits KB.20210609.06301.jpg KB.20210609.53963.fits.gz KB.20210609.06350.fits KB.20210609.53963.jpg KB.20210609.06350.fits.gz KB.20210609.54022.fits KB.20210609.06350.jpg KB.20210609.54022.fits.gz KB.20210609.06532.fits KB.20210609.54022.jpg KB.20210609.06532.fits.gz KB.20210609.54081.fits KB.20210609.06532.jpg KB.20210609.54081.fits.gz KB.20210609.06713.fits KB.20210609.54081.jpg KB.20210609.06713.fits.gz KB.20210609.54141.fits KB.20210609.06713.jpg KB.20210609.54141.fits.gz KB.20210609.06894.fits KB.20210609.54141.jpg KB.20210609.06894.fits.gz KB.20210609.54200.fits KB.20210609.06894.jpg KB.20210609.54200.fits.gz KB.20210609.07075.fits KB.20210609.54200.jpg KB.20210609.07075.fits.gz KB.20210609.54261.fits KB.20210609.07075.jpg KB.20210609.54261.fits.gz KB.20210609.07257.fits KB.20210609.54261.jpg KB.20210609.07257.fits.gz KB.20210609.54349.fits KB.20210609.07257.jpg KB.20210609.54349.fits.gz KB.20210609.07730.fits KB.20210609.54349.jpg KB.20210609.07730.fits.gz KB.20210609.54410.fits KB.20210609.07730.jpg KB.20210609.54410.fits.gz KB.20210609.08018.fits KB.20210609.54410.jpg KB.20210609.08018.fits.gz KB.20210609.54461.fits KB.20210609.08018.jpg KB.20210609.54461.fits.gz KB.20210609.08306.fits KB.20210609.54461.jpg KB.20210609.08306.fits.gz KB.20210609.54491.fits KB.20210609.08306.jpg KB.20210609.54491.fits.gz KB.20210609.19239.fits KB.20210609.54491.jpg KB.20210609.19239.jpg KB.20210609.54551.fits KB.20210609.19322.fits KB.20210609.54551.fits.gz KB.20210609.19322.fits.gz KB.20210609.54551.jpg KB.20210609.19322.jpg KB.20210609.54611.fits KB.20210609.19422.fits KB.20210609.54611.fits.gz KB.20210609.19422.fits.gz KB.20210609.54611.jpg KB.20210609.19422.jpg KB.20210609.54672.fits KB.20210609.19512.fits KB.20210609.54672.fits.gz KB.20210609.19512.fits.gz KB.20210609.54672.jpg KB.20210609.19512.jpg KB.20210609.54732.fits KB.20210609.19629.fits KB.20210609.54732.fits.gz KB.20210609.19629.fits.gz KB.20210609.54732.jpg KB.20210609.19629.jpg KB.20210609.54792.fits KB.20210609.21362.fits KB.20210609.54792.fits.gz KB.20210609.21362.fits 2 KB.20210609.54792.jpg KB.20210609.21362.fits.gz KB.20210609.54940.fits KB.20210609.21362.jpg KB.20210609.54940.fits.gz KB.20210609.21478.fits KB.20210609.54940.jpg KB.20210609.21478.fits.gz KB.20210609.55013.fits KB.20210609.21478.jpg KB.20210609.55013.fits.gz KB.20210609.21722.fits KB.20210609.55013.jpg KB.20210609.21722.fits.gz KB.20210609.55087.fits KB.20210609.21722.jpg KB.20210609.55087.fits.gz KB.20210609.23465.fits KB.20210609.55087.jpg KB.20210609.23465.fits.gz KF.20210609.03013.fits KB.20210609.23465.jpg KF.20210609.03013.fits.gz KB.20210609.24762.fits KF.20210609.03013.jpg KB.20210609.24762.fits.gz KF.20210609.20538.fits KB.20210609.24762.jpg KF.20210609.20538.fits.gz KB.20210609.26488.fits KF.20210609.20538.jpg KB.20210609.26488.fits.gz KF.20210609.20638.fits KB.20210609.26488.jpg KF.20210609.20638.fits.gz KB.20210609.27759.fits KF.20210609.20638.jpg KB.20210609.27759.fits.gz cal KB.20210609.27759.jpg logs KB.20210609.29417.fits plots KB.20210609.29417.fits.gz redux

(kcwidrp) rgrifft1@GriffsMacBook logs % more DRPF.log 2021-07-15 11:09:08:DRPF:INFO: Framework initialized 2021-07-15 11:09:09:DRPF:INFO: Framework main loop started 2021-07-15 11:09:10:DRPF:INFO: Event completed: name start_bokeh, action StartBokeh, arg name undef, recurr False 2021-07-15 11:09:12:DRPF:INFO: Event failed: name next_file, action ingest_file, arg name KB.20210609.21722.fits.gz, recurr True 2021-07-15 11:09:13:DRPF:INFO: Event failed: name next_file, action ingest_file, arg name KB.20210609.21722.fits.gz, recurr True

(kcwidrp) rgrifft1@GriffsMacBook logs % more KCWI.log 2021-07-15 11:09:08:KCWI:INFO:kcwi_proctab.py: proc table file not found: kcwi.proc 2021-07-15 11:09:09:KCWI:INFO:StartBokeh.py: Enabling BOKEH plots 2021-07-15 11:09:10:KCWI:INFO:kcwi_file_primitives.py: ------------------- Ingesting file KB.20210609.21722.fits.gz ------------------- 2021-07-15 11:09:12:KCWI:INFO:kcwi_file_primitives.py: <<< read 1 imgs and 1 tables out of 2 hdus in KB.20210609.21722.fits.gz 2021-07-15 11:09:12:KCWI:INFO:kcwi_proctab.py: Looking for MBIAS frames 2021-07-15 11:09:12:KCWI:INFO:kcwi_proctab.py: Camera is BLUE 2021-07-15 11:09:12:KCWI:INFO:kcwi_proctab.py: Target type is MBIAS 2021-07-15 11:09:12:KCWI:INFO:kcwi_proctab.py: Looking for frames with CCDCFG = 1101009 2021-07-15 11:09:12:KCWI:INFO:kcwi_proctab.py: Looking for CONTBARS frames 2021-07-15 11:09:12:KCWI:INFO:kcwi_proctab.py: Camera is BLUE 2021-07-15 11:09:12:KCWI:INFO:kcwi_proctab.py: Target type is CONTBARS 2021-07-15 11:09:12:KCWI:INFO:kcwi_proctab.py: Looking for frames with STATEID = 60c010bdce9e241c20199565 (Lenses) 2021-07-15 11:09:12:KCWI:INFO:kcwi_proctab.py: Looking for MFLAT frames 2021-07-15 11:09:12:KCWI:INFO:kcwi_proctab.py: Camera is BLUE 2021-07-15 11:09:12:KCWI:INFO:kcwi_proctab.py: Target type is MFLAT 2021-07-15 11:09:12:KCWI:INFO:kcwi_proctab.py: Looking for frames with STATEID = 60c010bdce9e241c20199565 (Lenses) 2021-07-15 11:09:12:KCWI:INFO:kcwi_proctab.py: Looking for INVSENS frames 2021-07-15 11:09:12:KCWI:INFO:kcwi_proctab.py: Camera is BLUE 2021-07-15 11:09:12:KCWI:INFO:kcwi_proctab.py: Target type is INVSENS 2021-07-15 11:09:12:KCWI:INFO:kcwi_proctab.py: Looking for frames with STATEID = 60c010bdce9e241c20199565 (Lenses) 2021-07-15 11:09:12:KCWI:INFO:kcwi_proctab.py: Looking for ARCLAMP frames 2021-07-15 11:09:12:KCWI:INFO:kcwi_proctab.py: Camera is BLUE 2021-07-15 11:09:12:KCWI:INFO:kcwi_proctab.py: Target type is ARCLAMP 2021-07-15 11:09:12:KCWI:INFO:kcwi_proctab.py: Looking for frames with STATEID = 60c010bdce9e241c20199565 (Lenses) 2021-07-15 11:09:12:KCWI:WARNING:kcwi_file_primitives.py: Cannot reduce OBJECT frame. Rescheduling for later. Found: 2021-07-15 11:09:12:KCWI:WARNING:kcwi_file_primitives.py: MBIAS: 0 :

[MNBrod]

I think the issue here is the use of kb*.fits, as opposed to KB*.fits, which is what matches the files you have.

Try running reduce_kcwi -f KB*.fits -g instead. Additionally, the pipeline will run smoother if you run it on a directory containing only uncompressed fits files, so I'd recommend removing the .gz files from the directory. Feel free to respond to this comment with your results!

[MNBrod]

Linked to #89

[rgriffiths1947]

(kcwidrp) rgrifft1@GriffsMacBook sci % reduce_kcwi -f KB.20210609.41370.fits -g 2021-07-22 15:32:48:KCWI:INFO: proc table file not found: kcwi.proc 2021-07-22 15:32:48:DRPF:INFO: Framework initialized 2021-07-22 15:32:48:DRPF:INFO: Framework main loop started 2021-07-22 15:32:48:KCWI:INFO: Enabling BOKEH plots 2021-07-22 15:32:48:DRPF:INFO: Event completed: name start_bokeh, action StartBokeh, arg name undef, recurr False 2021-07-22 15:32:48:DRPF:INFO: Event completed: name add_only, action add_to_dataframe_only, arg name KB.20210609.41370.fits, recurr False 2021-07-22 15:32:48:KCWI:INFO: ------------------- Ingesting file KB.20210609.41370.fits ------------------- 2021-07-22 15:32:49:KCWI:INFO: <<< read 1 imgs and 1 tables out of 2 hdus in KB.20210609.41370.fits 2021-07-22 15:32:49:KCWI:INFO: Looking for MBIAS frames 2021-07-22 15:32:49:KCWI:INFO: Camera is BLUE 2021-07-22 15:32:49:KCWI:INFO: Target type is MBIAS 2021-07-22 15:32:49:KCWI:INFO: Looking for frames with CCDCFG = 2201009 2021-07-22 15:32:49:KCWI:INFO: Looking for CONTBARS frames 2021-07-22 15:32:49:KCWI:INFO: Camera is BLUE 2021-07-22 15:32:49:KCWI:INFO: Target type is CONTBARS 2021-07-22 15:32:49:KCWI:INFO: Looking for frames with STATEID = 60c04c42ce9e241c20199566 (DESI) 2021-07-22 15:32:49:KCWI:INFO: Looking for MFLAT frames 2021-07-22 15:32:49:KCWI:INFO: Camera is BLUE 2021-07-22 15:32:49:KCWI:INFO: Target type is MFLAT 2021-07-22 15:32:49:KCWI:INFO: Looking for frames with STATEID = 60c04c42ce9e241c20199566 (DESI) 2021-07-22 15:32:49:KCWI:INFO: Looking for INVSENS frames 2021-07-22 15:32:49:KCWI:INFO: Camera is BLUE 2021-07-22 15:32:49:KCWI:INFO: Target type is INVSENS 2021-07-22 15:32:49:KCWI:INFO: Looking for frames with STATEID = 60c04c42ce9e241c20199566 (DESI) 2021-07-22 15:32:49:KCWI:INFO: Looking for ARCLAMP frames 2021-07-22 15:32:49:KCWI:INFO: Camera is BLUE 2021-07-22 15:32:49:KCWI:INFO: Target type is ARCLAMP 2021-07-22 15:32:49:KCWI:INFO: Looking for frames with STATEID = 60c04c42ce9e241c20199566 (DESI) 2021-07-22 15:32:49:KCWI:WARNING: Cannot reduce OBJECT frame. Rescheduling for later. Found: 2021-07-22 15:32:49:KCWI:WARNING: MBIAS: 0 2021-07-22 15:32:49:KCWI:WARNING: MFLAT: 0 2021-07-22 15:32:49:KCWI:WARNING: ARCLAMP: 0 2021-07-22 15:32:49:DRPF:INFO: Event failed: name next_file, action ingest_file, arg name KB.20210609.41370.fits, recurr True 2021-07-22 15:32:49:KCWI:INFO: ------------------- Ingesting file KB.20210609.41370.fits ------------------- 2021-07-22 15:32:49:KCWI:INFO: <<<

[MNBrod]

The pipeline is currently not able to figure out what other files it needs to go process in order to successfully reduce a science frame. That is to say, if the calibrations have not been processed, the reduction of a single science frame will fail (I will change the documentation to reflect that, failing to mention that was an oversight on our part).

I would recommend reducing the entire directory with the command I included above, which will reduce all of the calibrations and science frames. Once you reduce everything, you can go back and re-reduce any science frames using the command you tried above, and it should work. The key is that the calibrations have to have been processed first.

[rgriffiths1947]

Thanks, Max - that works!

[MNBrod]

Great to hear.