PyTorch Pruning with ProxylessNAS

[singagan]

Hi,

I am interested in using pruning with ProxylessNAS. So I added a new class like "MBInvertedConvLayer_Sparse" in ProxlyessNAS example that applies pruning in forward pass.

example:

 def forward(self, x):
        x = self.inverted_bottleneck(x)
        x = self.depth_conv(x)
        x = self.point_linear(x)
        return prune.ln_structured(nn.Conv2d, 'weight', 0.2)

However, I get the below error:

  File "~nni/retiarii/oneshot/pytorch/proxylessnas.py", line 74, in forward
    return ArchGradientFunction.apply(
  File "~nni/retiarii/oneshot/pytorch/proxylessnas.py", line 31, in forward
    return output.data
AttributeError: 'NoneType' object has no attribute 'data'

Any idea how I could hack proxylessNAS for it? Thanks!

[ultmaster]

Firstly please upgrade NNI to the latest version.

Secondly MBInvertedConvLayer_Sparse looks like a hack itself. And Proxyless is also a hack. I don't have confidence that these two hacks will work together.

[singagan]

Thanks. How do you mean "Proxyless is also a hack"? Is there any other NAS approach that might be better suited? Thank you!

[matluster]

"Proxyless is also a hack": ProxylessNAS is implemented by manipulating the backward pass, and did some forwards there. It has certain assumptions for the input and output format, which I'm not sure whether it will work with your space.

Is there any other NAS approach that might be better suited?

Maybe you can try ENAS.

Also I noticed that you are using the legacy NAS examples, which is long out-of-date. Please try the new examples.

[singagan]

@matluster Thanks for the quick reply. I see. Yes, I modified the input/output format to match my space. Thanks for pointing out the new examples. I am looking for a hardware-aware NAS. Maybe I can inject some hardware metrics in ENAS as well.

[Lijiaoa]

@singagan Could you close this issue as completed?