Stop after all configurations have been evaluated

[Bandd-k]

Does OpenTuner support option to stop after It launches all search configurations? For instance, I have three configurations int from 1 to 3. OpenTuner continues to work after it launches all configurations.

[jbosboom]

OpenTuner doesn't currently do this.

Most search spaces are far too large for OpenTuner to decide whether the space is exhausted by looking at the database of past configurations, so OpenTuner would have to compute the size of the space at the start, subtract 1 each time a configuration runs, and stop when the count reaches 0. Parameters have a search_space_size that returns an estimated search space size (and ConfigurationManipulator takes the product). The estimates are correct for most discrete parameters, but float-valued parameters always report 2 \ 32. Some parameters admit multiple representations of the same logical value, requiring normalization, such as ScheduleParameter; computing the exact size of the space of those parameters may not be tractable.

For spaces that are small enough for exhaustive search, we could add an exhaustive search technique that just iterates through the space trying all configurations, then stops requesting configurations. (I believe OpenTuner does stop if no configurations (not even duplicates) were requested for some number of generations.)

[jansel]

@jbosboom is correct. The use case OpenTuner is optimized for is for search spaces far too large for exhaustive search. Adding an exhaustive search technique would be a good (and easy) addition though. If you only have 3 configurations, the best strategy is to just try all 3 and stop.

[Bandd-k]

Thank You!