search

idstcv / ZenNAS

218 stars 35 forks source link

NAS-Bench-201 #15

Closed farhad-dalirani closed 2 years ago

farhad-dalirani commented 2 years ago

Hello,

As I remember from the paper, your method works on Vanilla CNN. However, in algorithm 1, you just mentioned that residual connections are deleted.

It confused me a little bit, and I do not know your method can be applied on any CNN without residual connection or it only can be applied on Vanilla CNN.

Can I use your code on benchmarks such as NAS-Bench-201?

dovedx commented 2 years ago

这是来自QQ邮箱的假期自动回复邮件。   邮件我已经成功接收，我会尽快处理并答复，谢谢！

MingLin-home commented 2 years ago

Hi Farhad, Thank you for your feedback!

Q1: Can you method be applied with residual link? A1: Yes and no.

For "Yes", because 1) a single-branch residual network can always be constructed by searching without res-link and then adding res-link afterward. 2) If you directly applied our method with residual link but pretended that there was no res-link, the final architecture is usually the same.

For "No", because our Zen-score, by its mathematical definition, requires that the layer is "linear scaling" w.r.t. its input. Residual link will destroy this linear property of CNN layers. So applying Zen-NAS to res-linked layers is not theoretically founded, although numerically doable.

Q2: Can Zen-NAS be applied on NAS-Bench-201? A2: As aforementioned in A1, it is not theoretically meaningful to apply Zen-score on non-VCNN blocks. Particularly, NAS-Bench-101/201/301 search spaces are highly non-regular. This makes them difficult to satisfy the "linear scaling" property. In practice, we found that the zen-score on NAS-Bench-X01 is not strongly correlated with accuracy.

Just my two cents. Please let me know if you have more questions!

Best, Ming

On Tue, Mar 22, 2022 at 7:06 PM Farhad Dalirani @.***> wrote:

Hello,

As I remember from the paper, your method works on Vanilla CNN. However, in algorithm 1, you just mentioned that residual connections are deleted.

It confused me a little bit, and I do not know your method can be applied on any CNN without residual connection or it only can be applied on Vanilla CNN.

Can I use your code on benchmarks such as NAS-Bench-201?

— Reply to this email directly, view it on GitHub https://github.com/idstcv/ZenNAS/issues/15, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFIQVWOLAE5ZFEB6Z4DQN7TVBJ4C5ANCNFSM5RMTAH5Q . You are receiving this because you are subscribed to this thread.Message ID: @.***>

farhad-dalirani commented 2 years ago

Thank you for your detailed reply. It is kind of you.

Sincerely, Farhad