Evaluations with respect to reference run

[ambros-gleixner]

In the aggregated Ipet table I would like to display

• number of instances faster/slower by a certain threshold
• create a filter group with all instances that have the same solving path as the default, heuristically identified by same number of nodes and LP iterations

I guess that both require a similar extension to how Ipet currently computes evaluations. I assign this to @GregorCH for now.

@fschloesser

[GregorCH]

This is not straightforward. There are two possible ways to achieve this:

1. Define a filter with a column, eg, LP iters, and use something like a base group check box to indicate that the particular value should be equal,unequal, larger, smaller, <=,>= to the default group.

   • Advantages: A base group always exists, it does not need to be declared by name (declaring a base group explicitly by name makes it harder to use the same evaluation on different data), such a filter syntax could be easily used for column member filter, as well.
   • Disadvantage: This approach is still subject to the one/any methodology. However, a filter group that requires all tests to be different from the default is empty by definition.

2. Enable to filter on the comparison columns themselves.

   • Advantage: The comparison (Q or D) columns are already computed and are available. The use of the comparison columns is straightforward, and implicitly respects the choice of the base group.
   • Disadvantage: The comparison columns are computed after the normal columns. This makes it harder to integrate them for column member filters. A second downside is that a change of the comparison type (quotient to difference, for example) must be changed in the filter attributes, as well, because those names are hardcoded.

[ambros-gleixner]

We just came up with the following solution:

3. New filter operator "isdifferent" and "isequal" (in addition to drop and keep).

Then an eval file to select instances with equal LP iterations would look like:

<Filter anytestrun="all" datakey="LP_Iters" operator="isequal">

[GregorCH]

I implemented this. The corresponding operators are "equal" and "diff", not "isequal" and "isdiff".