search

facebookresearch / SparseConvNet

Submanifold sparse convolutional networks
https://github.com/facebookresearch/SparseConvNet
Other
2.05k stars 333 forks source link

There seems a bug for 1x1 convolution #50

Closed LaiSongxuan closed 6 years ago

LaiSongxuan commented 6 years ago

Hello. I'm now using the SparseConvNet for character regconition. And I found when I set the filter size to 1 for the scn.Convolution layer, then the layer had empty output. I did not dive deep into the source code. Is this a bug? How can I fix it?

btgraham commented 6 years ago

Hmm. I have never actually tested sparseconvnet.Convolution with 1x1 filters.

For 1x1 convolutions, the sparsity pattern does not change, so it is better to use 'sparseconvnet.SubmanifoldConvolution' with size 1x1 or (even better) 'sparseconvnet.NetworkInNetwork' (equivalent to the above but optimized for size 1x1).

See for example https://github.com/facebookresearch/SparseConvNet/blob/6698676762805c1fb67c1840bdb77d7805ff2e2e/sparseconvnet/networkArchitectures.py#L144

LaiSongxuan commented 6 years ago

Thank you. : )

LaiSongxuan commented 6 years ago

Hello, @btgraham. I now have a ten-layer standard convolutional network, and the input is indeed sparse (only about 5% of the input is active; the input size is Nx7x200x400). However I found the pytorch implementation is several times faster than the sparseconvnet implementation. The latter has iterative SubmanifoldConvolution and BatchNormReLU layers. The GPU utilization is about 95% in both cases.

Is the performance gap normal (since pytorch uses high optimized cudnn libraries) or I have done something wrong?

btgraham commented 6 years ago

It will depend on the network, and how the sparsity changes through the network (due to convolutions, pooling, etc). I can take a look at the network, and a typical input, if you show me.

SparseConvNet is not as heavily optimised as PyTorch/CUDNN kernels, so for training relatively small networks running on the GPU, dense convolutions may be faster. Sparsity is more important in 3D, or when running on the CPU/low power hardware.

LaiSongxuan commented 6 years ago

The network is not deep, and does not change the sparsity pattern a lot. I have profiled the program, and it seems the convolution backward kernels are much slower than the pytorch kernels.

Can you please give me you email? I'll send you my code. I have cleaned and simplified the code, and it should be easy to run by "python main.py".

btgraham commented 6 years ago

btgraham@gmail.com

bigsheep2012 commented 5 years ago

@LaiSongxuan hello, may I know some details about your pytorch implementation that several times faster than sparseconvnet implementation?
You just use some combination of torch.nn.conv2d or torch.nn.conv3d layers in your pytorch implementation?

LaiSongxuan commented 5 years ago

@bigsheep2012 Right. I use iterative layers of nn.conv2d, nn.Batchnorm2d and nn.RELU.