search

automl / NASLib

NASLib is a Neural Architecture Search (NAS) library for facilitating NAS research for the community by providing interfaces to several state-of-the-art NAS search spaces and optimizers.
Apache License 2.0
528 stars 117 forks source link

Reasons not to use Optuna? #184

Closed tomaz-suller closed 4 months ago

tomaz-suller commented 5 months ago

Hi. I'm starting a new research project on NAS and found out about your library, which seemed really interesting but also seemed to overlap significantly with the more popular Optuna. I was curious to see if you had considered using it prior to developing your own library, and if so which drawbacks led you to develop your own tools.

In a previous research project also in NAS, we used Optuna since we didn't require much sophisticated control over the algorithms (though Optuna does allow for that with e.g. custom samplers), and we were very much implementation-oriented rather than research-oriented.

Having feedback will help a lot in using the most suitable tool for the job. Thanks in advance!

arberzela commented 5 months ago

hi @tomaz-suller, thanks a lot for your interest in NASLib. We haven't actually considered using Optuna when developing the NASLib, mainly for 2 reasons:

  1. As far as I can see, Optuna is more tailored to general purpose blackbox optimization algorithms, and with NASLib we aimed for a library that would be specific to NAS and provide support for performance predictors, various NAS search spaces, gradient-based and gradient-free NAS optimizers, etc.
  2. As you mentioned NASLib is much more research-oriented and it has been used in many papers published from our group. Therefore, we wanted flexibility in changing low-level functionalities required to conduct the necessary empirical evaluations in those research projects. This includes graph manipulations, encodings for different search spaces, etc. I would guess that these are all possible with Optuna as well, however we haven't investigated that further.

I would be interested to know your observations on the significant overlaps that NASLib has with Optuna.

Cheers, Arber

tomaz-suller commented 4 months ago

Thanks for your response. My comment really came from a too simplistic view of NAS, and in fact I didn't know about gradient-based or other approaches which didn't treat the problem as a black-box optimisation one.