microsoft / archai

Accelerate your Neural Architecture Search (NAS) through fast, reproducible and modular research.
https://microsoft.github.io/archai
MIT License
467 stars 92 forks source link

[BUG] some typos in algos.ipynb #157

Closed lovettchris closed 1 year ago

lovettchris commented 1 year ago

Describe the bug

  1. This doesn't work on windows:

    !ls ./out_evo

    But this does:

    %ls out_evo
  2. What does sinalized mean: # Used to sinalized if this evaluator is compute intensive.

  3. '# The evaluation cached is built' => perhaps "cache" is better here?

  4. Given most people use the dark theme, I wonder if we can improve the black on dark gray problem in the plots:

    image

  5. The MoBananasSearch search seems to have failed saying:

    2023-02-06 19:07:03,551 - archai.discrete_search.algos.bananas — WARNING —  No mutations found after 30 tries for each one of the 10 parents.
    2023-02-06 19:07:03,552 - archai.discrete_search.algos.bananas — INFO —  Found 0 new architectures satisfying constraints.
    2023-02-06 19:07:03,552 - archai.discrete_search.algos.bananas — INFO —  No new architectures found. Stopping search ...

    But I think this is saying "No new useful mutations found" and so it is choosing to stop and report what it found so far? I think the warning is misleading.

  6. It would be very interesting to compare the models found by each search. Their pareto curves look very similar, but are the model architectures the same of wildly different? MoBananasSearch found one bigger model with 0.989 accuracy that was twice as big, but everything else looks very similar in the range of latencies and accuracies. But some kind of conclusion blurb would be nice at the end of this notebook...

piero2c commented 1 year ago

Hi @lovettchris, thanks for the feedback.

I made a few changes to fix some of these issues: 1 - 3. Fixed typos

  1. Plot backgrounds are now white instead of transparent
  2. This message happens when the algorithm is not able to find any new architecture during the mutation step. It usually means that the search space is too small.
  3. I added a plot comparing pareto frontiers and a small comment. The results are pretty similar (probably also due to the fact the search space is too small)

It's currently on my to-do list to replace the search space used in this notebook with a larger and more meaningful one, this should improve points 5 and 6.