GregorCH / ipet

Interactive Performance Evaluation Tools for Optimization Software
MIT License
26 stars 6 forks source link

Inconsistent filtering of Cor@l test set #53

Closed ambros-gleixner closed 7 years ago

ambros-gleixner commented 7 years ago

When I use the attached evaluation file, e.g., to evaluate the runs in

https://rubberband.zib.de/result/AV6vdJBzysND7cYl8t1H?compare=AV7Tl7xuysND7cYl9Jps

then I get 212 instances using the hard-coded Cor@l filter, and 210 when I list the problem names individually. The number of instances is always 349 and looks identical. I cannot see where the 2 instances come from or which they are. (I only used it through Rubberband.) eval_coral.xml.txt

GregorCH commented 7 years ago

As I assumed, the two instances that are simply numbers are causing problems.

ipet-evaluate -t check.mipdev-solvable.scip-4.0.1.linux.x86_64.gnu.opt.spx2.none.M620v3-low.default.trn -f txt

diff ./Cor@al\ ipet\ indiv..txt Cor@al\ ipet.txt 
2a3,4
> 22433            1.15         4.0  1.000 1.000
> 23588            3.16       604.0  1.000 1.000

The filter engine of IPET tries to infer the type of a value by preferring integers over reals over strings. This problems does not occur for the inbuilt Coral filter, because it uses a hash set of suitable strings.

The solution for this problem is to use the column dtype for the value.

GregorCH commented 7 years ago

I will merge this as soon as the pull request succeeds.

ambros-gleixner commented 7 years ago

Thanks!