search

PyWiFeS / pipeline

The Python data reduction pipeline for WiFeS
6 stars 26 forks source link

run_derive_calib: poylfit poorly conditioned warnings #56

Open bmiszalski opened 2 months ago

bmiszalski commented 2 months ago

When running run_derive_calib on the test dataset classical_20230922 I get these warnings:

/usr/local/lib/python3.11/dist-packages/pywifes/wifes_calib.py:574: RankWarning: Polyfit may be poorly conditioned
  temp_best_calib = numpy.polyfit(temp_full_x, temp_full_y, polydeg)
/usr/local/lib/python3.11/dist-packages/pywifes/wifes_calib.py:604: RankWarning: Polyfit may be poorly conditioned
  best_calib = numpy.polyfit(full_x, full_y, polydeg)
/usr/local/lib/python3.11/dist-packages/pywifes/wifes_calib.py:607: RankWarning: Polyfit may be poorly conditioned
  best_calib = numpy.polyfit(full_x, full_y, polydeg)

It may be worth investigating these further.

Full output:

Deriving sensitivity function
cube_fn_list =['/code/WiFeS/classical_20230922/reduc_blue/OBK-124128-WiFeS-Blue-UT20230922T090650-1.p08.fits']
Found star HD200654
i_mid 1987 <class 'int'>
/usr/local/lib/python3.11/dist-packages/pywifes/wifes_calib.py:574: RankWarning: Polyfit may be poorly conditioned
  temp_best_calib = numpy.polyfit(temp_full_x, temp_full_y, polydeg)
/usr/local/lib/python3.11/dist-packages/pywifes/wifes_calib.py:604: RankWarning: Polyfit may be poorly conditioned
  best_calib = numpy.polyfit(full_x, full_y, polydeg)
/usr/local/lib/python3.11/dist-packages/pywifes/wifes_calib.py:607: RankWarning: Polyfit may be poorly conditioned
  best_calib = numpy.polyfit(full_x, full_y, polydeg)
best_calib
poly
None
[-5.24319378e-87  9.27170120e-83 -2.87821729e-79 -2.07717069e-75
  9.83936597e-73  5.56474206e-68  2.46468957e-64 -1.38920254e-61
 -7.50710660e-57 -4.19689941e-53 -5.38499094e-50  8.39401439e-46
  6.92422748e-42  2.04175926e-38 -6.94012109e-35 -1.02223388e-30
 -4.17392776e-27  5.54172444e-24  1.52346626e-19  6.02521323e-16
 -1.84368992e-12 -2.46812350e-08 -2.21053038e-06  7.87764729e-01
 -2.62218764e+03  2.60966260e+06]

run_derive_calib: All done in 0.0 seconds.
CMartinezLombilla commented 1 month ago

We've been exploring this issue and it's likely due to the high-degree polynomial used in the fitting (polydeg=30) in derive_wifes_calibration(). Fitting a polynomial of such a degree will likely lead to numerical instability and overfitting.