hayk-skydio
commented
1 year ago Hey @ameysutavani! Sorry for the very late reply on my part. We're super excited to have you contribute in this area. There are a lot of promising goals in the space of SymForce + PyTorch, and also more generally in the space of performant SLAM.
I believe the first bullet is the best place to start out, by building a differentiable SLAM example using SymForce to generate a PyTorch graph and on-manifold derivatives and then using it to learn the task end-to-end. @aaron-skydio just made a barebones PyTorch backend for generating free functions, which is the best starting point here. More codegen work is needed to generate the geometry and camera classes, which would make PyTorch usage much more convenient. We would be very happy for you to start down this path.
Beyond that, I'm also quite interested in ideas around parallelizing large-scale SfM or GBP. My knowledge consists mostly of methods that parallelize these methods on CPU, so I'd be curious if you have any references for doing GPU acceleration of sparse SLAM problems. Another exciting project would be to attempt a directly SLAM system using SymForce + PyTorch, that may map nicely onto the GPU.
Tagging people for additional thoughts: @xipeng-wang-skydio @ryan-skydio @zachteed
Hello all,
I wanted to start a discussion about SymForce + PyTorch project ideas and find use-cases from the community that might improve/enable new functionality in robotics applications.
For conversation starters, here are some of the ideas that I find interesting:
Taking inspiration from GradSLAM: https://gradslam.github.io/, it would be a nice experiment to add factor-graph based cost functions to a fully differentiable slam pipeline. Something like this can allow to fully flow the geometric residues and covariances all the way up to the feature extraction layers. Maybe adding custom geometric factors can lead to better convergences than chamfer or quadric losses. One could essentially learn features using factors as a training signal.
We can also take a look at some massive sfm problems that can benefit from parallelization on gpu. Pytorch can possibly be a (simpler) lower hanging fruit than going down the CUDA route.
We can also come up with a couple of slam + prediction problems that may benefit from a common graphical representation in torch. Something on the lines of prediction aided slam.
I am also curious about gaussian belief propagation: https://gaussianbp.github.io/ and it's parallels with factor graphs and transformers. Can be an interesting idea to leverage that to guide the gradient descent in slam and sfm problems.
I am very curious to learn about even more ideas and applications from the symforce community that would benefit from the above integration. Looking forward to gathering a pool of ideas here and then we can use this space to roadmap them according to feasibility, impact and added value.
CC: @hmartiro @aaron-skydio