Closed LaiSongxuan closed 6 years ago
Hmm. I have never actually tested sparseconvnet.Convolution with 1x1 filters.
For 1x1 convolutions, the sparsity pattern does not change, so it is better to use 'sparseconvnet.SubmanifoldConvolution' with size 1x1 or (even better) 'sparseconvnet.NetworkInNetwork' (equivalent to the above but optimized for size 1x1).
See for example https://github.com/facebookresearch/SparseConvNet/blob/6698676762805c1fb67c1840bdb77d7805ff2e2e/sparseconvnet/networkArchitectures.py#L144
Thank you. : )
Hello, @btgraham. I now have a ten-layer standard convolutional network, and the input is indeed sparse (only about 5% of the input is active; the input size is Nx7x200x400). However I found the pytorch implementation is several times faster than the sparseconvnet implementation. The latter has iterative SubmanifoldConvolution and BatchNormReLU layers. The GPU utilization is about 95% in both cases.
Is the performance gap normal (since pytorch uses high optimized cudnn libraries) or I have done something wrong?
It will depend on the network, and how the sparsity changes through the network (due to convolutions, pooling, etc). I can take a look at the network, and a typical input, if you show me.
SparseConvNet is not as heavily optimised as PyTorch/CUDNN kernels, so for training relatively small networks running on the GPU, dense convolutions may be faster. Sparsity is more important in 3D, or when running on the CPU/low power hardware.
The network is not deep, and does not change the sparsity pattern a lot. I have profiled the program, and it seems the convolution backward kernels are much slower than the pytorch kernels.
Can you please give me you email? I'll send you my code. I have cleaned and simplified the code, and it should be easy to run by "python main.py".
btgraham@gmail.com
@LaiSongxuan hello, may I know some details about your pytorch implementation that several times faster than sparseconvnet implementation?
You just use some combination of torch.nn.conv2d or torch.nn.conv3d layers in your pytorch implementation?
@bigsheep2012 Right. I use iterative layers of nn.conv2d, nn.Batchnorm2d and nn.RELU.
Hello. I'm now using the SparseConvNet for character regconition. And I found when I set the filter size to 1 for the scn.Convolution layer, then the layer had empty output. I did not dive deep into the source code. Is this a bug? How can I fix it?