It would be good to have a benchmark suite where you run an NxM test, graph properties x algorithm (x random seed). Then you use that to calibrate your algorithm selection, and also make some pretty plots to stick in the readme and convince people that the algorithm selector makes the right choices in these cases.
Sometimes it will not be fair. For example, algorithm selection for shortest path for a graph with signed edges will always pick (supposedly slower) algorithm that supports negative edges, even if the graph contains only positive edges, as a safe choice. But otherwise this is a good way how to validate the selection criteria.
Quoting this Reddit comment:
Sometimes it will not be fair. For example, algorithm selection for shortest path for a graph with signed edges will always pick (supposedly slower) algorithm that supports negative edges, even if the graph contains only positive edges, as a safe choice. But otherwise this is a good way how to validate the selection criteria.