Is there a way to query the best network architecture and accuracy

[linnanwang]

Hello,

This will be an improvement. Could you please suggest the simplest way to query the best model architecture and its accuracy? Thank you.

[chrisying]

Hi linnanwang,

There are multiple ways to define the "best model architecture" (e.g. highest validation, highest test, mean validation, mean test, etc...). Here's how I would find the best model using any of these metrics:

best_metric = -1        # higher = better
best_cell = None
for h in nasbench.hash_iterator():
  fixed, computed = nasbench.get_metrics_from_hash(h)
  metric = your_metric_function(computed)   # e.g. average test accuracy
  if metric > best_metric:
    best_metric = metric
    best_cell = (fixed['module_adjacency'], fixed['module_operations'])

best_spec = api.ModelSpec(best_cell[0], best_cell[1])

This should only take a few seconds (or a minute or so) to run.

For your reference, the best spec according to MEAN TEST ACCURACY is:

best_spec = api.ModelSpec([[0, 1, 1, 0, 0, 1, 1],
                           [0, 0, 0, 0, 0, 1, 0],
                           [0, 0, 0, 1, 0, 0, 0],
                           [0, 0, 0, 0, 1, 0, 0],
                           [0, 0, 0, 0, 0, 1, 0],
                           [0, 0, 0, 0, 0, 0, 1],
                           [0, 0, 0, 0, 0, 0, 0]],
                           [u'input', u'conv1x1-bn-relu', u'conv3x3-bn-relu',
                            u'maxpool3x3', u'conv3x3-bn-relu',
                            u'conv3x3-bn-relu', u'output'])

EDIT: correction to best_spec