More fixes in post-processing pipeline

[mkruckow]

Parts done/to do by @mkruckow:

• [x] Allow to search for a second termination code in case of min_timestep_limit, to over rule min_timestep_limit (see issue #254 )
• [x] There is now a general framework to add values from the histories at He depletion to the final_values.
  • Error during MODEL05 Patton&Sukhbold20-engineN20 core collapse prescrition! The history did not contain core masses at He depletion! (seen in step_3 of CO-HeMS_RLO; see question on slack #running_grids; c.f. issue #260 )
• [x] Take care of "unknown" termination flags
  • Some "unknown" resulted from the new interpolation class stable_reverse_MT, which were not recognized as stable_MT (see issue #284 )
  • Some "unknown" have been caused by having termination flag 2 stating initial MT, while termination flag 1 doesn't
  • Some "unknown" termination flags have been caused by using different thresholds, hence all limits and thresholds got spin off into utils/limits_thresholds.py
• [x] Use np.nan for lambda, where it is not well defined and set Ebind to 0 if lambda is not well defined. (see issue #279 ) Additionally, correct doc-string for CEE_parameters_from_core_abundance_thresholds and energy check.
• [x] Add more options in the pipeline ini files:
  • Add global switches to turn off all plots/checks.
  • Add cluster name to rerun section to be no longer inferred from the path.
• [x] Combining pipeline plots into image files with several pages (see issue #278 )
  • Allow the user to specify a plot file extension (make pdf new default)
  • Make pdfs multi-page plots if the extension is multipage-pdf.
• [x] Adding profile interpolation to the pipeline (see issue #213 )
  • It need some more refinement to get the profile interpolation working for all girds, but it is now available in the pipeline
• [x] Updating doc-strings and Errors in the pipeline
• Update tutorial in the documentation (may be skipped here because it is unclear what base data set to use for an independent user.)

Parts on the interpolators, hence feedback from @philipp-rajah needed:

• [x] see PR #295 :
  • UserWarning: y_pred contains classes not in y_true (seen in step_4 of CO-HeMS grid)
  • Not understood yet and looks to happen inside sklearn, hence may not be solved.
  • ValueError: All-NaN slice encountered (seen in step_4 of CO-HeMS_RLO)
  • Has a dirty workaround since PR #237. Is there a better solution without a try?
  • Mainly solved, there still appear some Warnings like .conda/envs/posydon-dev-p3_11/lib/python3.11/site-packages/numpy/lib/nanfunctions.py:1563: RuntimeWarning: All-NaN slice encountered or POSYDON/posydon/interpolation/IF_interpolation.py:968/971: RuntimeWarning: Mean of empty slice from time to time.
  • UserWarning: 1NN interpolation used for N binaries out of hull. popping up too often (seen in step_4_plotting of HMS-HMS and CO-HMS gird)
  • The warning should get extended by telling how far the most extreme one is outside the hull (if this output happens in normalized values, it needs to the scaling parameters).
  • Now it looks like this: 1NN interpolation used for 19 binaries out of hull. Parameter-wise distance (Unnormalized) for point with maximum out-of-hull euclidian distance (Normalized): [0.02182272 0.6391205 0.01501442].
  • RuntimeWarning: overflow encountered in divide (seen in step_4_plotting of CO-HMS gird)
  • Unclear how to solve, hence stays unsolved.
  • UserWarning: skip normalize: c=initial_MT, inds=[...] (seen in step_4_plotting of CO-HMS gird)
  • This warning is removed hence the skipping will happen in a silent way.

Testing

• [x] Testing done on CO-HMS gird (at one metallicity) and CO-HeMS grid
  • profile interpolation still has some issues -> will be taken care of in later PR
  • checking individual failing plots -> will be taken care of in later PR
  • not yet tested:
  • other girds (may done after PR is merged with new development branch then)

Open stuff:

• Get an updated tutorial for the pipeline. This needs first a solution to have a free available base data from some MESA runs (several GB).
• Enhance out of hull warning to provide relative errors, too.
• Understand open Warnings/Errors/Failures:
  • Interpolator (e.g. UserWarning: y_pred contains classes not in y_true, RuntimeWarning: overflow encountered in divide)
  • Profile interpolation
  • Individual failing plots

[astroJeff]

Update (Apr 11): Close to being finished by @mkruckow. Some tasks left to be completed by @philipp-rajah

[astroJeff]

Update (Apr 25): We discussed this PR. We do not want to accept until last sklearn/interpolation errors are resolved. @philipp-rajah to look into

[astroJeff]

Update (May 9): Still waiting on updates from Philipp. We need a couple bugs fixed before we can re-run the post-processing scripts.

[celiotine]

I reviewed the error messages in setup-pipeline and run-pipeline and everything looks good!