search

microsoft / nni

An open source AutoML toolkit for automate machine learning lifecycle, including feature engineering, neural architecture search, model compression and hyper-parameter tuning.
https://nni.readthedocs.io
MIT License
14.05k stars 1.82k forks source link

[Retiarii] migrate FBNet from NAS 1.0 to Retiarii framework #3814

Open QuanluZhang opened 3 years ago

QuanluZhang commented 3 years ago

What would you like to be added: migrate FBNet from NAS 1.0 to Retiarii framework

Why is this needed: NAS 1.0 will be deprecated

Without this feature, how does current nni work：still works for now

Components that may involve changes: the NNI Retiarii part

AnkitRai-22 commented 3 years ago

I want to work on this. I have a doubt regarding which repo subfolder should I push the code files into. Currently we have the fbnet located in nni/nni/algorithms/nas/pytorch/fbnet but since we are shifting it to retiarii should I create a new folder with all the code files inside the nni/nni/algorithms/ ? Please let me know at the earliest.

QuanluZhang commented 3 years ago

@AnkitRai-22 thanks for your contribution. the new version of fbnet should be put under this folder https://github.com/microsoft/nni/tree/master/nni/retiarii/oneshot/pytorch . feel free to discuss with us if you have any other question.

AnkitRai-22 commented 3 years ago

Should I continue discussion through comments or do we use gitter.im or discord for this?

QuanluZhang commented 3 years ago

Should I continue discussion through comments or do we use gitter.im or discord for this?

we can discuss in this issue, and schedule discussion meeting when necessary

AnkitRai-22 commented 3 years ago

I went through the other implementations of oneshot nas models using retiarii and had a few queries which I wanted to get cleared. Lemme know when you are available for a discussion meeting. @ultmaster if you could join it would be great

QuanluZhang commented 3 years ago

I went through the other implementations of oneshot nas models using retiarii and had a few queries which I wanted to get cleared. Lemme know when you are available for a discussion meeting. @ultmaster if you could join it would be great

you can send me your email address, so that we can send you a teams meeting invitation

ultmaster commented 3 years ago

I think FBNet is nothing but a simple gumbel softmax. The implementation should look very much similar with ProxylessNAS or DARTS. You can copy ProxylessNAS implementation (in Retiarii of course) as your template and do some modifications, and this is done!

It's likely that there will be a refactor to oneshot algorithms in the near future, after which, this task should be more simple. If you prefer to wait, you can wait (probably 3 months).

AnkitRai-22 commented 3 years ago

My email is ankitrai22march@gmail.com and thanks @ultmaster for replying. I will go through both the implementations and let you know if I need some help.

AnkitRai-22 commented 3 years ago

There is a class named MixedOp in mutator.py for proxyless at line https://github.com/microsoft/nni/blob/a606916513ad1da36cc606f26f1c349ddca7d6fe/nni/algorithms/nas/pytorch/proxylessnas/mutator.py#L36, how did you transform it in the proxyless.py file in retiarii framework. @ultmaster

AnkitRai-22 commented 3 years ago

If you could explain it would be great

ultmaster commented 3 years ago

The old implementation of ProxylessNAS mixed op looks excessively complicated, and I don't suggest you look into details of this file.

The more simple way to accomplish the goal is to understand what is going on in the implementation of Retiarii's implementation of ProxylessNAS. I'm sure it will be much easier for you.

AnkitRai-22 commented 3 years ago

@ultmaster I just wanted to know the work-around for the mutable parameter which was being passed around in the MixedOp class, since this class is present in FBNet too. 'mutable' is mentioned as a LayerChoice function, so did you replace the MixedOp with ProxylessLayerChoice Class in the retiarii implementation?

AnkitRai-22 commented 3 years ago

@ultmaster If you are comfortable can we link up through a voice channel? Discord, google meet, microsoft teams, anything is fine with me. Just have a lot of queries and it would be great, if we could link up and get them cleared in short amount of time.

ultmaster commented 3 years ago

What are your available time slots?

AnkitRai-22 commented 3 years ago

I am comfortable with any time slot

ultmaster commented 3 years ago

How about ~20 minutes later? I'll send you a meeting link.

AnkitRai-22 commented 3 years ago

Sure

ultmaster commented 3 years ago

I've sent a link. Please check your email to see whether it's available for joining.

AnkitRai-22 commented 3 years ago

Yeah, it's available

AnkitRai-22 commented 3 years ago

I've joined it, did you cancel it?

ultmaster commented 3 years ago

There are two meeting links. Sorry for mistake. Please try another one.