search

GregorCH / ipet

Interactive Performance Evaluation Tools for Optimization Software
MIT License
26 stars 6 forks source link

Test IPET behavior if instance order is inconsistent #65

Closed GregorCH closed 6 years ago

GregorCH commented 6 years ago

between out and err file.

fschloesser commented 6 years ago

IPET seems to be very confused: 

>> cat scip-short-scrambled.out |grep @01
5:@01 /instances/CP/linking.cip ===========
327:@01 /instances/MIP/blend2.mps ===========
714:@01 /instances/MIP/dcmulti.mps ===========
1129:@01 /instances/MIP/egout.mps ===========
1478:@01 /instances/MIP/enigma.mps ===========
1823:@01 /instances/MIP/flugpl.mps ===========
2180:@01 /instances/MIP/gt2.mps ===========
>> cat scip-short-scrambled.err |grep "\(code\|@01\)"
2:@01 /instances/CP/linking.cip ===========
6:returned with error code 1
9:@01 /instances/MIP/bell5.mps ===========
13:returned with error code 2
16:@01 /instances/MIP/egout.mps ===========
19:returned with error code 3
22:@01 /instances/MIP/flugpl.mps ===========
28:returned with error code 4
31:@01 /instances/MIP/enigma.mps ===========
35:returned with error code 5
Instancewise Results:
GitHash     5e0f46f                  
               Time Status  Err TimeQ
ProblemName                          
bell5         24.27     ok  2.0 1.000
egout         18.96     ok  3.0 1.000
enigma         4.50     ok  5.0 1.000
flugpl         0.10   okok  4.0 1.000
gt2            0.53     ok  NaN 1.000
linking        0.25     ok  1.0 1.000

Do you have an idea?

GregorCH commented 6 years ago

The error file should be considered as supplemental information. If the order is not acceptable, we could simply skip the remainder of this error file and print a warning message. Regarding missing instance logs in the out file that are present in the error file, I would suggest to add a replacement and warn the user that something went missing.