Particle swarm optimization (PSO) is a simple and computationally cheap optimization method by exploring the search space with a coordinated swarm. Since most available tuning strategies only generate a list of models without any optimization heuristics, PSO would be a valuable addition to the existing collection of strategies.
For our purposes of hyperparameter tuning, a good implementation should be able to support both NumericRanges and NominalRanges, or any combinations of the two. To this end categorical hyperparameters would be embedded as probability vectors to extend usual PSO variants.
A naive algorithm would expose some meta-hyperparameters to the user (e.g. inertia, cognitive and social coefficients), but an adaptive scheme such as in Optim.jl could automate this choice.
Given the population of the swarm, each agent maps to a model to be evaluated at each iteration. The total number of iterations is calculated from the set number of models n of the TunedModel. In cases where the strategy generates a surplus of models (e.g. 3 agents and a maximum number of 10 models would leave 2 models unevaluated in the last iteration), then the extra models would be saved for later warm-starts (see #131 for more detail).
Particle swarm optimization (PSO) is a simple and computationally cheap optimization method by exploring the search space with a coordinated swarm. Since most available tuning strategies only generate a list of models without any optimization heuristics, PSO would be a valuable addition to the existing collection of strategies.
For our purposes of hyperparameter tuning, a good implementation should be able to support both
NumericRanges andNominalRanges, or any combinations of the two. To this end categorical hyperparameters would be embedded as probability vectors to extend usual PSO variants.A naive algorithm would expose some meta-hyperparameters to the user (e.g. inertia, cognitive and social coefficients), but an adaptive scheme such as in Optim.jl could automate this choice.
Given the population of the swarm, each agent maps to a model to be evaluated at each iteration. The total number of iterations is calculated from the set number of models
nof theTunedModel. In cases where the strategy generates a surplus of models (e.g. 3 agents and a maximum number of 10 models would leave 2 models unevaluated in the last iteration), then the extra models would be saved for later warm-starts (see #131 for more detail).