POLAR crashes when it comes to continuum analysis

[larnoldgithub]

Occuring when the routine deals with the continuum analysis:

--OBS_DIR: /data/spirou4/apero-data/offline/red/23BQ01-Aug09
19:04:20.672-**|POLAR[11892]|    --EXPOSURES[0]: [TELLU_OBJ] 2890757o_pp_e2dsff_tcorr_AB.fits
19:04:20.689-**|POLAR[11892]|    --EXPOSURES[1]: [TELLU_OBJ] 2890758o_pp_e2dsff_tcorr_AB.fits
19:04:20.707-**|POLAR[11892]|    --EXPOSURES[2]: [TELLU_OBJ] 2890759o_pp_e2dsff_tcorr_AB.fits
19:04:20.748-**|POLAR[11892]|    --EXPOSURES[3]: [TELLU_OBJ] 2890760o_pp_e2dsff_tcorr_AB.fits

it's TOI4616

19:04:38.828-**|POLAR[11892]| ***************************************************************************
19:04:38.889-**|POLAR[11892]|I[40-021-00014]: Part 3: Run continuum analysis
19:04:38.927-**|POLAR[11892]| ***************************************************************************
19:04:38.965-  |POLAR[11892]|A[40-021-00003]: Calculating Stokes I total flux
19:04:39.255-  |POLAR[11892]|A[40-021-00024]: Calculating polarization using IRAF
19:04:39.517-!!|POLAR[11892]|E[01-010-00001]: Unhandled error has occurred: Error <class 'ValueError'>
19:04:39.533-!!|POLAR[11892]|
19:04:39.550-!!|POLAR[11892]|Traceback (most recent call last):
19:04:39.567-!!|POLAR[11892]|  File "/apero/apero-drs/apero/core/utils/drs_startup.py", line 433, in run
19:04:39.584-!!|POLAR[11892]|    llmain = func(recipe, params)
19:04:39.601-!!|POLAR[11892]|  File "/apero/apero-drs/apero/recipes/spirou/apero_pol_spirou.py", line 158, in __main__
19:04:39.617-!!|POLAR[11892]|    pprops = gen_pol.calculate_continuum(params, recipe, pprops)
19:04:39.634-!!|POLAR[11892]|  File "/apero/apero-drs/apero/science/polar/gen_pol.py", line 1447, in calculate_continuum
19:04:39.651-!!|POLAR[11892]|    contpol = _fit_continuum(params, recipe, wldata, pol,
19:04:39.667-!!|POLAR[11892]|  File "/apero/apero-drs/apero/science/polar/gen_pol.py", line 2600, in _fit_continuum
19:04:39.684-!!|POLAR[11892]|    spl = interpolate.splrep(wavemap[mspec], spec[mspec], k=3,
19:04:39.701-!!|POLAR[11892]|  File "/conda/miniconda3/envs/apero-env/lib/python3.9/site-packages/scipy/interpolate/_fitpack_py.py", line 288, in splrep
19:04:39.718-!!|POLAR[11892]|    res = _impl.splrep(x, y, w, xb, xe, k, task, s, t, full_output, per, quiet)
19:04:39.734-!!|POLAR[11892]|  File "/conda/miniconda3/envs/apero-env/lib/python3.9/site-packages/scipy/interpolate/_fitpack_impl.py", line 508, in splrep
19:04:39.751-!!|POLAR[11892]|    raise _iermess[ier][1](_iermess[ier][0])
19:04:39.768-!!|POLAR[11892]|ValueError: Error on input data
19:04:39.787-!!|POLAR[11892]|

@edermartioli wrote in an email: Indeed this crash should not occur. I've had this error in the past, and it is related to the interpolation inside the continuum detection routine. This is a routine that I got from J. Morin that imitates IRAF's task to detect continuum. Anyway, there is a way to possibly fix it without changing the codes. You may change the parameters, which I'm not sure where is located in APERO as Neil imported my code and refactored it to fit APERO framework. Below is an extract of how these parameters look like in my codes. You may change the continuum fit function from 'spline3' to 'polynomial' and then you may also change the "ORDER" from a high value (e.g. 50 -> number of knots for spline) to something like 8 if you're using polynomial. By the way, if you just decrease the number of knots and keep spline it may also get rid of the error. These errors I believe is when the interpolation splits the spectrum into several chunks using the knots and some spectral ranges are empty, so we end up with an error as one cannot define the interpolation within that range.

I hope that helps.

cheers, Eder

   #### Parameters below are for 'IRAF' algorithm :
    #function to fit to the polar continuum: must be 'polynomial' or
'spline3'
    if p['IC_POLAR_CONT_POLYNOMIAL_FIT'] :
        p['IC_POLAR_IRAF_CONT_FIT_FUNCTION'] = "polynomial"
    else :
        p['IC_POLAR_IRAF_CONT_FIT_FUNCTION'] = "spline3"
    #function to fit to the stokes I continuum: must be 'polynomial' or
'spline3'
    #p['IC_STOKESI_IRAF_CONT_FIT_FUNCTION'] = "spline3"
    p['IC_STOKESI_IRAF_CONT_FIT_FUNCTION'] = "polynomial"
    # polar continuum fit function order: 'polynomial': degree or
'spline3': number of knots
    p['IC_POLAR_IRAF_CONT_FUNCTION_ORDER'] =
p['IC_POLAR_CONT_DEG_POLYNOMIAL']
    # stokes I continuum fit function order: 'polynomial': degree or
'spline3': number of knots
    p['IC_STOKESI_IRAF_CONT_FUNCTION_ORDER'] = 50
    #p['IC_STOKESI_IRAF_CONT_FUNCTION_ORDER'] = 8
    # plot Polarimetry continuum calculation using IRAF algorithm
    p['IC_POLAR_IRAF_CONT_PLOT'] = False
    # plot Stokes I continuum calculation using IRAF algorithm
    p['IC_STOKESI_IRAF_CONT_PLOT'] = False
    #################################

[larnoldgithub]

Thanks Eder, I finally could have the polar routine working.

1/ edit your /apero/apero-drs/apero/core/instruments/spirou/default_constants.py 2/ go near the end of the file, line 2723 and change POLAR_IRAF_CONT_FUNC_ORDER.value = 20 to POLAR_IRAF_CONT_FUNC_ORDER.value = 15

It worked also with 10 on my test polar sequence: apero_pol_spirou.py 23BQ01-Aug09 --exposures 2890757o_pp_e2dsff_tcorr_AB.fits 2890758o_pp_e2dsff_tcorr_AB.fits 2890759o_pp_e2dsff_tcorr_AB.fits 2890760o_pp_e2dsff_tcorr_AB.fits --crunfile=run_CCF_to_end.ini --program=POLAR[11892] --recipe_kind=polar-tcorr --shortname=POLAR --parallel=True

[larnoldgithub]

@edermartioli @CharlesCadieux @clairem789

[njcuk9999]

Just a note:

1. Do not change the default_constants.py - we should do that at the github level if this is a permanent change (@edermartioli is this a permanent change for all files? Changing this will not allow you to update from github!!!
2. If you want to change it "forever" on your machine change the user_constants.ini file
3. If you want to change it just for one file or a set of files you can add it to a custom run.ini file and run that with apero_processing