Profile analysis routine

[paulstapor]

Profile likelihood results are not necessarily easy to interpret. Yet, having worked with them for a while, one can read a lot out of bad results. It would be really, really cool to have a routine such as "analyze_profiling_results", which checks profile likelihood results, pinpoints possible issues, and suggests (based on stored optimization options) changes in the options for each profile. Ideally, a second routine could be run which just applies these change and reruns profiling, in order to get better results... This routine could give scores to computed profiles, based on how many too high angles there are in a profile, how many drops in posterior ratio, how many optima were found, how many parameters run into bounds, etc... This could furthermore be nicely integrated with CI: i.e., changes to pyPESTO should not decrease the computed profile quality scorein an integration or a unit test (rather integration test to have something meaningful).

One far day, when I really have time, I would be happy to implement this... Only 36 years left to retirement...

[yannikschaelte]

Only 36 years left to retirement...

Sorry to break the news, but by that time retirement age must be >75 at least already.

Agreeing with the other points. At least a "analyze my profiles, tell me what is identifiable, or where potential flaws are" would be good.

[paulstapor]

I would imagine such a routine to do the following: Follow a profile and check:

• How deep are drops between profile points?
  • Give bad score for deep drops
  • If deep drop found: Check whether maybe some parameter has run into bounds
  • If deep drop found: Check jumps in parameter path, suggest other guessing scheme or tighter optimization tolerances
• How high is the angle in the posterior ratio profile?
  • Give bad score if many high angels (profile is zig-zagging)
  • If zig-zagging: Check jumps in parameter path, suggest other guessing scheme or tighter optimization tolerances
• Measure total variation in posterior ratio profile (ideally not exceeding one)
  • re-check for zigzagging closely, if total variation is high
• Is the path going up again?
  • Suggest rerunning from other local optimum, in not yet done
  • Check whether point better than global optimum found. Smash user in face if so, suggest rerunning optimization (give really bad score to profile, suggest running from new global optimum)

...And so on... Who wants to implement it? :D

[paulstapor]

Ah, right: And finally plot profile with red highlighting of the bad segments, potentially with clickable comments...