search

microsoft / nni

An open source AutoML toolkit for automate machine learning lifecycle, including feature engineering, neural architecture search, model compression and hyper-parameter tuning.
https://nni.readthedocs.io
MIT License
14.06k stars 1.82k forks source link

How to customize metric in multi-trial NAS #4250

Open ding3820 opened 3 years ago

ding3820 commented 3 years ago

Hi, Is it possible to incorporate a callback function for customized metric in multi-trial NAS like what one-shot NAS does? There's an argument metrics in one-shot NAS trainer such as DARTS, but I could not find where to implement this function for multi-trial NAS. What am I looking for is the other metric like AUC for the NAS algorithm to optimize. It seems to me that the default metrics for classification is acc and this might not be the best metric to optimize the model.

ultmaster commented 3 years ago

In that case, I think you need to inherit the class called Classification and do the customization.