search

facebookresearch / SparseConvNet

Submanifold sparse convolutional networks
https://github.com/facebookresearch/SparseConvNet
Other
2.05k stars 333 forks source link

Is it possible to find a way to make all computations on GPU ? #158

Closed tchaton closed 4 years ago

tchaton commented 4 years ago

Dear All,

I was wondering if there would be an way to make all computations on GPU, or at least speeding up the computation, as it is currently a bottleneck for trying out more complex models.

Best regards, Thomas Chaton.

btgraham commented 4 years ago

What do you mean by more complex models? How does the run time change when model complexity increases?

tchaton commented 4 years ago

Dear @btgraham,

Currently the sparseconv model I am training takes 30 min / epoch on my custom data. It converges in 5 days with pretty good results. I have tried more complex architectures, but I am getting training time around 2 h / epoch, which is too much.

However, I was wondering if you had any idea on how to make SpareConv faster ? And if a complete GPU based implementation would be possible ?

My current work can be found there: https://github.com/nicolas-chaulet/deeppointcloud-benchmarks. Have a look ! SparseConv / Minkowski Engine seems to scale 10 times better than KPConv, RandLA-Net, etc....

Best, Thomas Chaton.

tchaton commented 4 years ago

Hey @btgraham,

I am wondering if you have ideas on how to overcome the current bottleneck of this framework to make training / inference faster, scale to millions of points( 1M, 10M, 100M maybe) with the lowest memory consumption.

Best regards, Thomas Chaton.

btgraham commented 4 years ago

You can use checkpointing to reduce memory usage during training. i.e. take a section of the network and put it in a CheckpointedSequential unit. https://github.com/facebookresearch/SparseConvNet/blob/master/sparseconvnet/sequential.py i.e take a stack of residual units and wrap them.