Multi-Objective-NAS / self-supervised-nas

Official implementation of the paper "Pretraining Neural Architecture Search Controllers with Locality-based Self-Supervised Learning" (NeurIPSW 2020)

5 stars 0 forks source link

Locality(edit-distance vs. accuracy) #21

Closed Naruu closed 3 years ago

Naruu commented 3 years ago

20 - Does edit-distance meaningful for performance prediction?

[ ] Choose N samples at random. Find 1 samples with edit-distance=k for each sample. Get performance differences.
[ ] graph
- Figure with (1, N) subplots.
- y-axis : performance difference, x-axis : ratio of samples / count of samples
- type : CDF? histogram?

Naruu commented 3 years ago

issues

edit-distance calculation is expensive. Time consuming, cpu intensive.
N=? k=?
- NASBENCH101 has 423k unique graphs -> then N=?.
- Is k=3 enough?. Should it be up to 4 or 5?

currently N=300, k=5 watching how long it takes.