Experiment list

[juice500ml]

• [ ] Does edit-distance meaningful for performance prediction?
• [ ] Can edit-distance be learned?
• [ ] Is learning edit-distance useful for NAS task?

Does edit-distance meaningful for performance prediction?

NAS-Bench-101, locality, correlation between graph edit distance and the result performance. We check our edit-distance generation algorithm indeed results in correlation on performance.

• Choose 5000 samples at random. Find 1 samples with edit-distance=n for each sample. Get performance differences.
• CDF: x-axis: performance difference, y-axis: ratio of samples that is less different than the given difference.

Can edit-distance be learned?

We sample two graphs from the NAS-Bench-101 and determine whether the distance can be indeed predicted. (With multiple metric learning methods)

• Use edit-distance pairs from above.
• Generate sequence for each samples, output feature vectors.
• Calculate distance between feature vectors. (It differs depending on metric learning algorithm)
• x-axis: True edit-distance, y-axis: distance learned by metric learning algorithm

Is learning edit-distance useful for NAS task?

• [ ] Cover different seq length.
• [ ] Find hyperparameters for each metric learning algorithm:
  • [ ] TripletMargin
  • [ ] Angular
  • [ ]
  • [ ]
• [ ]

[Naruu]

Does edit-distance meaningful for performance prediction?

Wrote code but takes tooooooo long time It took 2.5 hours to find edit-distance=1,2,3,4 architecture from 5 seed architectures. Trying to use multiprocessing library, but still it would take massive amount of time 😢