0.7.289 PP_SCI does preprocess CAL files

[larnoldgithub]

The sequence pp_seq_opt documentation says

PP_CAL will only preprocess the calibration files PP_SCI will only preprocess the science files (non telluric hot star files)

but I see PP_SCI processing calibration files.

My ini file has RUN_PP_CAL = True RUN_PP_SCI = True

and id00000 = pp_seq_opt id00001 = calib_seq id00002 = science_seq

and with PP_SCI processing calibration files, CAL are preprocessed twice.

[njcuk9999]

Yes PP_SCI works with SCIENCE_TARGETS keyword - if you don't set this it will process all files (nothing is telling APERO what a "science file" is in the header - the only files filtered out are those with hot star object names. I'm not sure if it will mess things to add "CALIB" to this (for times when we want to preprocess everything).

PP_CAL works on objects with OBJECT="CALIB"

If you know of a separate header key that defines "science" that might make it easier

[larnoldgithub]

the fits kw SBCALI_P is P4 for calibrations and P5 for science (o.fits) would this help ?

SBCALI is the position of the calibration wheel in the Cass Unit.

[njcuk9999]

We may be able to add a key to the PP_SCI for SBCALI=P5 (so PP_SCI will only reduce files with SBCALI=P5). Would this be your expected behavior of PP_SCI (to process all SBCALI=P5 files)?

[larnoldgithub]

yes PP_SCI preprocessing science files means for me all observation files o.fits, and no calibrations included.

[njcuk9999]

Seem there are some unwanted files for SBCALI_P = P5

i.e.

+------------+---------------+------------+-------------------+
| KW_OBJNAME | KW_DPRTYPE    | KW_CALIBWH | COUNT(KW_DPRTYPE) |
+------------+---------------+------------+-------------------+
| SKY        | DARK_DARK_SKY | P5         |              1313 |
| CALIB      | DARK_DARK_TEL | P5         |              2924 |
| SKY        | DARK_FP_SKY   | P5         |                25 |
| CALIB      | OBJ_HCONE     | P5         |                 8 |
| CALIB      | OBJ_HCTWO     | P5         |                 1 |
| CALIB      | Unknown       | P5         |                 5 |
+------------+---------------+------------+-------------------+

Maybe I can do it a different way - though just cutting it down to these+science is better than all the calibrations for sure.

[njcuk9999]

Thinking about it I can just do:

DPRTYPE = ['POLAR_FP', 'POLAR_DARK', 'OBJ_DARK', 'OBJ_FP']

that should work for "science" observations

[larnoldgithub]

Seem there are some unwanted files for SBCALI_P = P5

i.e.

+------------+---------------+------------+-------------------+
| KW_OBJNAME | KW_DPRTYPE    | KW_CALIBWH | COUNT(KW_DPRTYPE) |
+------------+---------------+------------+-------------------+
| SKY        | DARK_DARK_SKY | P5         |              1313 |
| CALIB      | DARK_DARK_TEL | P5         |              2924 |
| SKY        | DARK_FP_SKY   | P5         |                25 |
| CALIB      | OBJ_HCONE     | P5         |                 8 |
| CALIB      | OBJ_HCTWO     | P5         |                 1 |
| CALIB      | Unknown       | P5         |                 5 |
+------------+---------------+------------+-------------------+

The 2 first lines are CALIBrations actually, as well as the 4th and 5th (and 6th), so should be P4.

The third one is a o.fits when we do a sky_before or a sky_after in an OG, like when we want to have a sky to know the signature of the persistence in a faint object observation. for example for the sky we did on Jul17 3028565o.fits we have for that science observation SBCALI_P= 'P5' which is what is expected.

[larnoldgithub]

Thinking about it I can just do:

DPRTYPE = ['POLAR_FP', 'POLAR_DARK', 'OBJ_DARK', 'OBJ_FP']

that should work for "science" observations

I would say yes it should work. If you modify only one file for this update, I will cp it from gituhb and test it.

[larnoldgithub]

and what about just taking o.fits without any kw for PP_SCI ?

[larnoldgithub]

sorry P5 was not a good solution. we do turn the calib wheel to P5 for one pair of the darks of daily calib set.

[njcuk9999]

and what about just taking o.fits without any kw for PP_SCI ?

The way the code is set up it just looks at header keys. I don't want to have to recode or hard code something like this when I can use header keys. My DPRTYPE header key works though so this is an easy fix.