[Question] Parallel evaluations with Ask-Tell

[TheEimer]

Using the ask-tell interface, we can in principle execute multiple configurations in parallel. Do you have an intuition if there are "safe" ways of doing this (e.g. ask for all configurations of a bracket and then tell them before switching brackets or doing the initial design fully and then limit parallelism to 10% of total trials at a time)? Do you expect a strong effect s.t. it would make sense to benchmark this or do you think this is probably not very relevant?

(I can do the benchmarking myself if you point me to a few good problems for it)

[benjamc]

• The initial design is safe to 100% parallelize.
• For the parallel sequential search we need to balance asking for configs and retraining the model to propose better configs. I have no intuition about how far you can push it, especially in the case of MF. 10% sounds reasonable. I would benchmark this using the yahpo MF set in carps. You can also check for batch BO literature. Unfortunately, we do not have a batch acquisition function in SMAC yet.