BlueBrain / BluePyOpt

Blue Brain Python Optimisation Library
https://bluepyopt.readthedocs.io/en/latest/
Other
198 stars 96 forks source link

New stagnation stopping criterion #450

Closed AurelienJaquier closed 1 year ago

AurelienJaquier commented 1 year ago

For reference, here is a description of the new stopping criterion: The problem we had with the previous stopping criterion I proposed was that, due to noise in the optimisation best fitness curve, it sometimes triggered too soon. This is why I decided to combine the same stopping criterion, stop if the median of the last 20 values of the best fitness has improved less than 1% over the last 100 generations, with another one, stop if the standard deviation of the best fitness across the last 20 generations is below a threshold of 2% of the best fitness. This prevents the first criterion to trigger due to noise, since the noise will prevent the second criterion to trigger. Combining these two criteria will greatly decrease the number of false positive.

codecov-commenter commented 1 year ago

Codecov Report

Merging #450 (7f8df7d) into master (5e4b75e) will decrease coverage by 0.40%. The diff coverage is 82.75%.

@@            Coverage Diff             @@
##           master     #450      +/-   ##
==========================================
- Coverage   87.87%   87.47%   -0.40%     
==========================================
  Files          50       50              
  Lines        4288     4312      +24     
==========================================
+ Hits         3768     3772       +4     
- Misses        520      540      +20     
Impacted Files Coverage Δ
bluepyopt/deapext/CMA_SO.py 95.65% <ø> (ø)
bluepyopt/deapext/optimisationsCMA.py 81.75% <0.00%> (ø)
bluepyopt/deapext/stoppingCriteria.py 81.75% <84.61%> (-12.60%) :arrow_down:
bluepyopt/deapext/CMA_MO.py 83.63% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

AurelienJaquier commented 1 year ago

Don't merge this yet, I was required to allow the user to not use the stagnation criterion if he/she wants to.

DrTaDa commented 1 year ago

Looks nice ! I still feel it stops a bit early in some cases but I am not sure I would be able to do better "by hand".

AurelienJaquier commented 1 year ago

@DrTaDa are you ok with the last changes I made? Is it ok to merge this?

DrTaDa commented 1 year ago

All good for me ! Go for it