search

D-X-Y / AutoDL-Projects

Automated deep learning algorithms implemented in PyTorch.
MIT License
1.56k stars 281 forks source link

Why retrain the last searched config of architecture by default instead of the best? #35

Closed Hanson13 closed 4 years ago

Hanson13 commented 4 years ago

Hi Xuanyi, thanks for your excellent work and your code is really convenient to use! I noticed that in the scripts-search folder of TAS, the scripts use seed-${rseed}-last.config to retrain the last searched architecture by default, but I am a little confused why not use seed-${rseed}-best.config to retrain the best searched architecture, which I think may has better performance? Which searched architecture did the result reported in your paper retrain, the last or best? Thank you!

D-X-Y commented 4 years ago

It is because the "best-config" is generated based on the validation accuracy per epoch. It might not align with the constraint of the computation cost. However, the last-config usually aligns with this constraint.

Hanson13 commented 4 years ago

but in https://github.com/D-X-Y/NAS-Projects/blob/master/exps/search-shape.py#L188 At the end of search, you save the best-config only when it meets the constraints, so I think the "best-config" is generated only after search and should always aligns with the constraint?

D-X-Y commented 4 years ago

Yes, in this case, it should align with the constraint. Another reason that does not choose "best-config" is that the validation accuracy is not the accuracy of a particular architecture derived from the current architecture parameters. It is the average accuracy on the validation set and each iteration might use different architecture according to the sampling. So the validation accuracy in the code can not be used to judge how good or how bad of a particular architecture.

Hanson13 commented 4 years ago

Thanks, I see your point. But I am still a little confused that "the last-config usually aligns with this constraint", I didn't see constraints when saving last-config in https://github.com/D-X-Y/NAS-Projects/blob/master/exps/search-shape.py#L180. Is that because it's at the end of optimization and we suppose it to satisfy the constraint?

D-X-Y commented 4 years ago

Yes, we suppose it should satisfy the constraint. Empirically, it does in most cases. Even if not, the computation cost of the-last-config is still very close to the target cost.

D-X-Y commented 4 years ago

Feel free to re-open it if you want.