simonprovost
commented
1 year ago EDIT 1.0: I will provide a more detailed update later; for now, it is time to take the win for today. I am not entirely certain that everything is functioning properly, but I am currently, at the time of writing this down, able to perform an 8-core multi-processed SMAC BO run with the GAMA async component, and it appears to function quite well.
There is a lot much more work to be done, and on various aspects, but I am now confident that a POC is feasible; within a week, I will be able to elaborate whether I have a fully workable POC. Therefore, if this POC seems to be promising, over the summer we would have a fully generalised BO's search algorithm for the GAMA's community.
In the interim, it would be preferable to await , if you planned answering, the more in-depth update later in the day-or-so; In a nutshell, I have taken a slightly different route from time to time than the one(s) proposed above, but the overall plan remains kind of the same anyway 💪
Cheers,
PGijsbers
Hi @PGijsbers! Hope you are doing okay!
In reference to #201, I would like to initiate this new Github issue with a more concrete proposal for a new search algorithm, namely Bayesian Optimisation, that must adheres to the GAMA design philosophy.
Prior to delving into the proposal in detail, and in accordance with Issue #201, I would like to point out that I tested and researched SMAC. Initially, we resolved some issues with the Authors through a pull request. Now, SMAC with a CASH problem is operating flawlessly. The final step in the interim is to implement this out-of-the-box framework via GAMA.
Therefore, to address the issue, on top-of-all I would like to propose a solution that not only integrates SMAC in a way that adheres to the standard basic guidelines outlined in the SMAC's README but also blends it harmoniously with GAMA's framework, ensuring we uphold GAMA's multiprocessing procedures. This approach is chosen for three critical reasons. First, multiprocessing, as seen in GAMA's async evaluator, stands as a pivotal aspect of GAMA's process. Second, preserving this functionality ensures that any future enhancements to GAMA would only necessitate minimal alterations to any currently and soon implemented search algorithms. Third, the optimize call, as detailed in the SMAC's README, executes its operation internally, leaving us without direct control. By opting for one of SMAC's flexible interfaces (proposed below), we can effectively manage the task without spawning multiple SMAC instances or inhibiting our implementation from being multiprocessing.
Consequently, retrieve below a preview, step-by-step implementation to ensure we are in compliance at the very least with what's below:
BO File Creation: This entails the very-basic creation of a new BO file similar to the pattern content in the GAMA's Random Search algorithm.
Dynamic Defaults Modification: The objective here is that through the Dynamic Default method, we translate/convert GAMA's search space into SMAC's automatically. Converting the search space from GAMA to SMAC should not present many difficulties. Nonetheless, we anticipate potential complexities when handling GAMA's search space param check component, also known as conditions in (SMAC) ConfigSpace. Therefore, this translation due to varying conditions in ConfigSpace and GAMA's param verification may present an obstacle. In order to provide flexibility, rather than overcomplicating the transformation's process, I propose enabling users to alter the search space via a parameter during the initialisation of the BO search algorithm. I.e, If there are no conditional relationships between hyperparameters and the user has not overridden the ConfigSpace search space, a translation from GAMA to SMAC (ConfigSpace) would be performed automatically. Nevertheless, if the user desires a more complex search space (e.g, conditions between hyperparameters), an override of this one could be transmitted to the BO's search algorithm instantiation, but will be kept internally for SMAC's framework to continue functioning flawlessly; GAMA has no need for the ConfigSpace. As a result, this strategy creates a balance between user control and algorithmic effectiveness.
Search Process Decomposition: The search process implementation will involve several steps:
_Note that here will consider utilising the Ask&Tell SMAC interface, which allows us to request a candidate setting based on an internal Bayesian optimisation process, and then evaluate ourselves the candidate in order to communicate the results back to the SMAC created instance. Further example of this interface will be available at the bottom of this post._
Instantiate Necessary Objects: The first step involves easily instantiating all the objects needed for SMAC's interface to function correctly.
Loop with SMAC.ask() Call: We enter a loop,
while (max_evaluations is None) or (len(output) < max_evaluations):, similar to what RandomSearch or ASHA of GAMA are doing. Within this loop, we first put all thefuture = operations.wait_next(async_) # and IF conditions to append the result in case future is nonecalls, but most importantly we run theSMAC.ask(), which provides the next candidate setting for evaluation.Bayesian Optimisation Evaluation Function: Following this, we need to invoke an
async_.submit()procedure such as what RandomSearch is doing. As first arg, will be put an internal function to the BO's search algorithm, namedbayesian_optimisation_evaluateusing same parameter design as howevaluate_pipelineis (here and here). Our internal function should performs three core tasks:SMAC.ask()provided.operations.evaluate(with the acquired parameters from thebayesian_optimisation_evaluate) with the just-created individual pipeline, and let GAMA do the evaluation and all the stuff it internally proceeds with.operations.evaluate(potentially those specifics will be helpful), and finally updates SMAC's instantiation usingSMAC.tell(). This ensures that for futureSMAC.ask()calls, the algorithm proposes the most efficient individual based on Bayesian optimisation principles.On the other hand, on second arg of the
async_.submit(), instead ofoperations.individual()that for exemple RandomSearch() does, we simply put the acquired Configuration from SMAC.ask() forbayesian_optimisation_evaluateto do its dedicated job as aforementioned.As a result, the new implementation will adhere to GAMA's design, utilise SMAC's
Ask & Tellinterface, and if more than one core is available and specified by the user, the feature will execute in a single SMAC's instance but will be multi-processed thanks to GAMA's dedicated component, so theSMAC.ask()will be called as multiple as the GAMA allows and theSMAC.tell()will updated as much as GAMA will ask it to do. Therefore, the BO's search algorithm will be multi process but by GAMA async evaluator object and not SMAC itself, plus will save all individuals and results for GAMA post-processing execution without much difference than whatRandomSearch()orASHA()do.While this serves as an initial proof-of-concept implementation, I thought about it a lot before writing all that, but note that more will be refined over the summer to incorporate meta-learning into SMAC at the initialisation and hopefully, a pull request will be submitted for review by mid/end of the summer. I.e, I first need to make something work with my design then I will take time to generalise it well for GAMA's community.
Request for Feedback
I would immensely value feedback and direction regarding this implementation workplan. Ie., Is there anything am I missing in regards to GAMA's internal process? any possible steps I should review further? I am not attempting to get you to adhere to the SMAC's API process because you do not know about it. Rather, I am focusing on my approach to implementing this and whether or not we at least addressed all of the GAMA's points.
Additional References
To assist in understanding the concepts and frameworks referenced, here are some helpful resources:
Thank you so much for your time. I know I am asking a lot of the GAMA team these days, but to make sure we are all on the same page, I am trying to do as well as you did for the GAMA community so that in the future the community can have a plethora of options that are all designed in the same way 💪
Wishing you a wonderful Sunday evening! Best wishes,