How can we know the accuracy of the best found architecture by trainer.export()?

[curiousjit]

Short summary about the issue/question:

In NAS examples (ENAS or DARTS), the final architecture can be exported by trainer.export(). However I do want to know the best found architecture's accuracy too. Is there any way I can get that? Or can I get a summary at the end of the best found architecture's accuracy after all the epochs have been completed?

Brief what process you are following:

I have used trainer.export() in ENAS and DASRTS (search.py). I have checked the trainer codes (ENAS DARTS) but I cannot find a way to do it.

It's in section External Repositories and References/External Repositories

How to reproduce it: In ENAS example, add trainer.export() at the end and run search.py nni Environment:

nni version: 1.5
nni mode(local|pai|remote): local
OS: Linux
python version: 3.6.10
is conda or virtualenv used?: Conda
is running in docker?: No

need to update document(yes/no): no

Anything else we need to know: no

[ultmaster]

Hi. Thanks for reaching out. For algorithms like ENAS and DARTS, search phase is only a search phase. There is no way to tell the architecture performance, except you train from scratch. So to obtain the final accuracy, please follow retrain procedure and train the architecture sufficiently.

[curiousjit]

Thank you clarifying it. Following your DARTS retraining example, I can get a good accuracy after retraining the best architecture. Is there a way to export this model after retraining phase? I know we can export models by using callback(ModelCheckpoint). Although I am not sure if I can do the same in the retraining code to export my final model.

[ultmaster]

Sure. You might notice that in retraining, we do a standard pytorch training. So you can follow standard pytorch model checkpoint by using state_dict and torch.save.

[curiousjit]

Got it! Thank you.