[ISSUE] Text bugs

[dharabor]

Suggest to change the text on the banner.

Currently: "A repository maintains the results of MAPF benchmarks" Proposed: "A database for benchmark results in Multi-Agent Path Finding"

The following are bugs in the text descriptions of plots.

1. (Best solutions plot, y-axis description) "For instances where no solution is reported, no algorithm can achieve the best solution in such cases."

I think we're trying to say we don't record any data for instances with no known solutions. But this is irrelevant for understanding the plot right? Like, the total number of instances is X, and every algorithm has a number of best instances <= X, regardless of whether solutions are known for every X.

My suggestion is to omit the text.

2. (Closed instances plot) "The unbounded-suboptimal and bounded suboptimal algorithms are ignored as they cannot close any instance."

This might be less a bug and more an issue for discussion. What does "Instances closed" refer to? I can see two possibilities.

(i) the subset of instances for which we know the optimal costs. Under this definition we should include in the plot all algorithms which can match the optimal cost.

(ii) the subset of instances where authors report a solution cost and a matching lower bound value. Under this definition we include a narrower set of algorithms.

I think at the moment we use definition (ii). One complicating issue is that it is possible for instances to be closed by a pair of algorithms working together. Like, an optimal method claims a best known bound equal to C, but it does not claim any plan. Another solver claims a best known solution with cost equal to C, but it does not claim any lower-bound. The instance is closed, yet according to the current filtering method, it does not appear in this plot (at least, that's how I interpret the situation).

3. (Best lower bound, description). "The number of instances achieving the best lower bound reflects the availability of optimal and bounded-suboptimal algorithms for proving optimality (i.e., higher the better). The purpose of this plot is to compare these algorithms and identify challenging scenarios. The unbounded-suboptimal algorithms are ignored as they do not report lower bounds."

Suggest to change the first sentence to say only "Higher is better" -- or merge just the brackets text with the previous sentence. The second sentence should say "challenging instances", not "challenging scenarios" (since we look at the data by #agents). The third sentence should say "Algorithms that do not report lower-bound data are omitted from this plot".

4. (Best lower bound, y-axis). "For instances where no lower bound is reported, no algorithm can achieve the best lower bound in such cases."

This bug is the same as (1).

5. (Instances solved, description). "The figure compare between different algorithms and identify challenging scenarios."

The sentence should say "challenging instances", not "challenging scenarios" (since we look at the data by #agents).

6. (Best lower bound, description, scenario level). "This plot compares the suboptimality gap (i.e., lower the better) between the lower bounds reported by MAPF algorithms w.r.t. the best-known lower bound. The figure is intended to compare the quality of lower bounds reported by different MAPF algorithms as the number of agents increases."

Suggest to reword as "This plot compares the quality of lower bounds reported by different MAPF algorithms, as the number of agents increases."

7. (Best lower bound, y-axis, scenario level). "The suboptimality is computed as (LB - LB') / LB, where LB' is the lower bound reported by each individual algorithm, and LB is the best-known lower bound. The y-axis shows the suboptimality ratio. "0%" indicates that the algorithm has reported the best lower bound, while "Inf" indicates that the algorithm has reported no lower bound."

Suggest to reword as:

"The y-axis shows the percentage gap between a reported lower-bound from a particular solver vs. the best known bound for that problem. The gap is computed as (LB - LB') / LB, where LB' is the reported lower bound and LB is the best-known lower bound. Lower is better. Note that the best known bound always has a gap of zero percent."

Suggest also to change the y-axis label from "Suboptimality ration" to "Gap to Best LB"

[bshen95]

Addressed all of them, please check.

[dharabor]

Most issues fixed. Still no resolution to Issue #2, which merits further discussion. Closing.