search

GregorCH / ipet

Interactive Performance Evaluation Tools for Optimization Software
MIT License
26 stars 6 forks source link

Wrong classification of instances as better #70

Closed ambros-gleixner closed 6 years ago

ambros-gleixner commented 6 years ago

Two issues, I am not sure:

  1. In the MINLP evaluation of the release report, some solved_not_verified instances were classified as better (-> limit), because of a bug having to do with the =bestdual= status in the solu file.

  2. I just encountered the same in this MIP comparison: https://rubberband.zib.de/result/AWBY7plxU7LMof4FxoM9?compare=AWFres7tqjghvac026R1#summary 4 instances (e.g.csched07) are also marked as better instead of solved_not_verified.

Suspicion: Do we need to reimport the old runs?

@fschloesser

fschloesser commented 6 years ago

@ambros-gleixner Yes reimporting could help, but some of the recent changes in ipet are not yet in rubberband, as this may require some restructuring, i am not sure yet how much. First we have to merge the pull requests into the ipet master.

ambros-gleixner commented 6 years ago

OK, let me know when it's done.

GregorCH commented 6 years ago

Have you actually tried reimporting the runs? Actually, the status codes better and solved not verified should not even occur as Status in IPET, nor will they be with the changes introduced in #67.

GregorCH commented 6 years ago

The most likely reason for the status better is that the solu file used during parsing was out of date. The four instances have been added to the solu file about a week before your upload.

With MR #67, all of them have the correct status "ok" in the mentioned run (thanks for the download feature, @fschloesser ).

GregorCH commented 6 years ago

Again, this is a strong argument for the changes introduced in #67, where reference information is reused (statusses are recomputed) at evaluation time. One may even go as far as to recompute the status information every time before the static rubberband table (in the tab "Details") is displayed.

GregorCH commented 6 years ago

@ambros-gleixner , The changes have been merged to the master. The mentioned problems do not occur anymore with updated reference data and the most recent IPET code. If you encounter similar problems, feel free to reopen this ticket or a new one. You can also comment whether we want the better/solved not verified status codes, or not.