search

microsoft / nni

An open source AutoML toolkit for automate machine learning lifecycle, including feature engineering, neural architecture search, model compression and hyper-parameter tuning.
https://nni.readthedocs.io
MIT License
14.05k stars 1.82k forks source link

PyTorch Pruning with ProxylessNAS #5231

Closed singagan closed 1 year ago

singagan commented 2 years ago

Hi,

I am interested in using pruning with ProxylessNAS. So I added a new class like "MBInvertedConvLayer_Sparse" in ProxlyessNAS example that applies pruning in forward pass.

example:

 def forward(self, x):
        x = self.inverted_bottleneck(x)
        x = self.depth_conv(x)
        x = self.point_linear(x)
        return prune.ln_structured(nn.Conv2d, 'weight', 0.2)

However, I get the below error:

  File "~nni/retiarii/oneshot/pytorch/proxylessnas.py", line 74, in forward
    return ArchGradientFunction.apply(
  File "~nni/retiarii/oneshot/pytorch/proxylessnas.py", line 31, in forward
    return output.data
AttributeError: 'NoneType' object has no attribute 'data'

Any idea how I could hack proxylessNAS for it? Thanks!

ultmaster commented 2 years ago

Firstly please upgrade NNI to the latest version.

Secondly MBInvertedConvLayer_Sparse looks like a hack itself. And Proxyless is also a hack. I don't have confidence that these two hacks will work together.

singagan commented 1 year ago

Thanks. How do you mean "Proxyless is also a hack"? Is there any other NAS approach that might be better suited? Thank you!

matluster commented 1 year ago

"Proxyless is also a hack": ProxylessNAS is implemented by manipulating the backward pass, and did some forwards there. It has certain assumptions for the input and output format, which I'm not sure whether it will work with your space.

Is there any other NAS approach that might be better suited?

Maybe you can try ENAS.

Also I noticed that you are using the legacy NAS examples, which is long out-of-date. Please try the new examples.

singagan commented 1 year ago

@matluster Thanks for the quick reply. I see. Yes, I modified the input/output format to match my space. Thanks for pointing out the new examples. I am looking for a hardware-aware NAS. Maybe I can inject some hardware metrics in ENAS as well.

Lijiaoa commented 1 year ago

@singagan Could you close this issue as completed?