Weird Plotting Ablation results

[mlindauer]

I tried to run it on some data from cplex on regions200. The stdout output looks reasonable -- except the last flip:

INFO:Ablation:Round  0 winner(s): (['lpmethod'], -12.8369)
INFO:Ablation:Round  1 winner(s): (['mip_cuts_mircut'], -13.2227)
INFO:Ablation:Round  2 winner(s): (['mip_limits_cutsfactor'], -6.7214)
INFO:Ablation:Round  3 winner(s): (['barrier_ordering'], -5.7829)
INFO:Ablation:Round  4 winner(s): (['simplex_limits_singularity'], -4.2256)
INFO:Ablation:Round  5 winner(s): (['simplex_pricing'], -4.0615)
INFO:Ablation:Round  6 winner(s): (['preprocessing_relax'], -3.0112)
INFO:Ablation:Round  7 winner(s): (['simplex_tolerances_markowitz'], -3.0432)
INFO:Ablation:Round  8 winner(s): (['mip_cuts_gubcovers'], -1.0924)
INFO:Ablation:Round  9 winner(s): (['mip_limits_aggforcut'], -0.6727)
INFO:Ablation:Round 10 winner(s): (['mip_cuts_zerohalfcut'], -0.6727)
INFO:Ablation:Round 11 winner(s): (['mip_limits_gomorycand'], -0.3744)
INFO:Ablation:Round 12 winner(s): (['mip_strategy_bbinterval'], -0.1675)
INFO:Ablation:Round 13 winner(s): (['preprocessing_dual'], -0.0622)
INFO:Ablation:Round 14 winner(s): (['mip_strategy_dive'], -0.0093)
INFO:Ablation:Round 15 winner(s): (['emphasis_mip'], 0.0021)
INFO:Ablation:Round 16 winner(s): (['mip_strategy_probe'], 0.0026)
INFO:Ablation:Round 17 winner(s): (['mip_cuts_cliques'], 0.0035)
INFO:Ablation:Round 18 winner(s): (['mip_cuts_implied'], 1.2320)
INFO:Ablation:Round 19 winner(s): (['barrier_limits_growth'], 3.9069)
INFO:Ablation:Round 20 winner(s): (['network_pricing'], 13.5587)
INFO:Ablation:Round 21 winner(s): (['mip_limits_submipnodelim'], 137.2505)

It would be strange if the last flip would increase the performance by 137.

Furthermore, the plot does not reflect the decrease in running time after the first flips; see attachment.

You can find my data here: /home/lindauer/projects/smac3/smac3-output_2016-11-25_13\:12\:17/

[AndreBiedenkapp]

I end up with a similar result. Seems like the target is predicted to result in a runtime worse than the source configuration. My first guess is that I somehow messed up with the EPARX predictions.

[AndreBiedenkapp]

Okay I just checked what happens when I use the standard SMAC model and unlog the mean predictions, which confirms my guess from before. I'll have to rethink the EPARX part.

[AndreBiedenkapp]

I used the wrong cutoff/thresholds for the EPARX predictions. Since I unlog everything I have to use the unloged cutoff and PAR values. Will be fixed soon