GregorCH / ipet

Interactive Performance Evaluation Tools for Optimization Software
MIT License
26 stars 6 forks source link

Incorrent primal integral result for SCIP #46

Closed stephenjmaher closed 7 years ago

stephenjmaher commented 7 years ago

I am finding that the primal integral values reported by ipet are incorrect. The issue seems to be related to when the First Solution is captured. This adds the first solution to the end of the primal bound list. When the primal integral is calculated, this is supposed to be appended to the start of the list. However, this does not occur.

I don't understand the getProcessPlotData function, so I am unable to find a fix for this solution. The current fix is to just ignore the table entry "First Solution". By ignoring this entry, an good approximation of the primal integral is calculate.

GregorCH commented 7 years ago

During lunch, @leoneifler pointed exactly this issue to me, thank you for pointing it out.

The file that needs modification is ipet/parsing/Solver.py, where SCIP appends the first solution to the list, instead of inserting it at the beginning. The inclusion of the first solution line causes problems, anyway, because its numerical precision for the time and bound is different from what the regular, periodic table reports.

GregorCH commented 7 years ago

Removing the first primal bound is, however, not clever: If a heuristic, usually heur_trivial, finds a solution even before presolving starts, this is not reflected in the normal table.

stephenjmaher commented 7 years ago

I know that is not a good solution, but do you know of a quick fix that I can use? In my case the presolving does not take too long, so there will not be too much lost by ignoring the first solution.

GregorCH commented 7 years ago

Give me another 5 minutes, I also need this for my own work.

stephenjmaher commented 7 years ago

Great.

GregorCH commented 7 years ago

Solved by a) Stripping the time precision and b) inserting the first solution only, if it is not present, at the beginning of the history.

stephenjmaher commented 7 years ago

Great, thanks for the fix

GregorCH commented 7 years ago

Hi Steve, I am now facing serious troubles and will devise a better way.

GregorCH commented 7 years ago

Hi Steve, I fixed this now correctly (I suppose). Thanks to this fix, my results look much better now. As an additional check, I could visually compare the average gap at termination based on the PrimalBoundHistory with the average Primal Gap in my tables (computed without histories). They are exactly the same.

stephenjmaher commented 7 years ago

Thanks for the fix. I will now test it out.

stephenjmaher commented 7 years ago

Looks like it works correctly for me too.

GregorCH commented 7 years ago

Are your results also better, now? I can report a primal integral reduction of more than 30%

stephenjmaher commented 7 years ago

The results look similar to my hacked fix, the results look similar. The primal integrals I was getting before the fix were nonsensical. Now they look like what I would expect after the computing the primal integral by hand.