need examples of where one hand washes the other

[timm]

It turns out that SBSE has much to offer MSR:

• hyper-parameter optimization of data miners (ICSE'16, IST'16) or data pre- processors (Amrit, ICSE'18)
• novel solutions to hard configuration problems (sven, ase15 using CART, some of vivek's work since then)
• direct application of user criteria to problems
• multi-objective EAs can be used for problems with multiple learning objectives (TOSEM'13; ICSE'16)
• EAs can be used to learn readable models (eg. some of the work on EAs for SEE)

further, MSR has much to offer SBSE particularly in the area of landscape analysis, surrogates, and explanation.

• SBSE tools walk over a landscape of potential solutions. If MSR tools are run before SBSE, then the shape of the landscape can be learned. With knowledge of this shape, SBSE tools can jump to better solutions, faster (e.g. GALE)
• SBSE tools need to evaluate new solutions which means that, if the evaluation process is slow, then the SBSE tool can become very slow indeed. An alternate approach, which is gaining popularity, is use the examples to date to build a surrogate model that can quickly guess the objective scores of a new example. Such surrogates may be somewhat inaccurate but, in practice, the cost of those inaccuracies are far out-weighed by the benefits of very fast evaluation (eg. zuluga, gaussian process models; Vivek's flash; GUI power consumption optimisation)
• SBSE solutions are often small points in a space of possibilities. While useful, such points can be overly-specific and brittle to an unknown degree (i.e.it is not known how much each point can be varied without losing the efficacy of the solution). MSR tools, on the other hand, are often volumes (not points) within the space of possibilities (e.g. the clusters found by K-means, or the space of examples accepted by a lead node of a decision tree). That is, MSR tools can generalize SBSE solutions. (e.g. Emmanuel Letier re'11 work)
• ML can learn the best operators to use in SBSE (e.g., Leandro's work on Information Sciences'18 and PPSN'16)

[markuswagnergithub]

Potentially off-topic, but I'll mention two alternatives to the classical surrogate anyway: 1) "incomplete information", as used in (https://link.springer.com/article/10.1007%2Fs00453-015-0103-x): 1.1) in partial evaluation of solutions, only a small amount of information about the problem is revealed in each fitness evaluation (e.g. to decide which of two solutions is better, a small and randomly chosen sub-problem is used (same for both)) 1.2) in partial evaluation of populations, not all individuals in the population are evaluated (read: either evaluated, or simply carried along) --> it turns out that (on the benchmarks), very little information is needed for decent runtimes 2) The deep learning people do this all the time, e.g. by considering each of millions of pictures just ones while doing to massive deep learning thing.

[timm]

1.2) in partial evaluation of populations, not all individuals in the population are evaluated (read: either evaluated, or simply carried along)

this is the core of the active learning results. @vivekaxl routinely eats problems that (offically) have 10^4 to 10^6 combinations with < 100 evaluations

[markuswagnergithub]

"active learning" is now on my reading list, as it must differ from Bayesian optimisation and the likes, and from what some educators use in class-rooms.

[timm]

"active learning" is now on my reading list, as it must differ from Bayesian optimisation

i agree the term is broader than BO but is not bayesian parameter optimzation and example of AL?

[markuswagnergithub]

Yeah,... not clear. It seems to depend on whom you ask. I've read up on BO long time ago... wayyy too long ago. I need to revisit this after the deadlines.

[timm]

is there room for this in the paper? or is it out-of-scope?

[markuswagnergithub]

I'd say it is out-of-scope for now. This is already quite an overview of the most popular methods that a small (but elite) group has objectively put together.