Jacobian access in python wrapper

[shteren1]

Feature

Hi I propose to have the jacobians of all gtsam calculations accessible through the python wrappers.

Motivation

Having jacobian access in python will help with testing a debugging challenging optimization from python, but most of all, this will improve the custom factors building from python considerably, currently one cannot use other gtsam calculation inside a custom factor, for instance projecting a 3d point onto a camera cannot be easily used in a python custom factor because there is no way to get the jacobians updated by the projection operation.

Pitch

All gtsam routines that accept optional jacobians will acccept numpy matrices in the correct shape from the python wrappers

Alternatives

Since some gtsam routine support getting a pointer for a non initialized jacobian matrix and initializes it within this might not propagate well back into python, an alternative can be some intermediate wrapper function that collects the jacobian from the cpp routine and returns it as an output. for instance: point2, Hcamera, Hlandmark = camera.Project(landmark, return_jacobians=True)

Additional context

I'm aware that it might be super complicated to pull off but this feature can make gtsam a very strong tool for the robotics and ML community working with python, being an alternative for other tools like JAX or pytorch for some type of optimization problems.

[dellaert]

It is indeed not easy I think. Just a note: even in Python you can take any NonLinearFactor (or nlfGraph) and call linearize to get its (whitened) Jacobians at a linearization Values instance. That’s how Yusuf and I found the NaNs for the essentialMatrix issue in the user group yesterday…

[dellaert]

Although, maybe it is easy. @varunagrawal, do you know what happens if we just “expose” an overload with the Jacobians typed as matrix in the .i files? There is some possibility it might just work :-)

[ProfFan]

It is not easy, but solvable. I already have a prototype that works (which will need an overload in the classes). Reason is pybind11 only does write back when the type is Eigen::Ref. Otherwise it results in a copy and the C++ changes to the Jacobians will be discarded.

[ProfFan]

@shteren1 @dellaert @varunagrawal You can try the demo in #1171.

[dellaert]

@ProfFan I see, very cool! The problem of course is that we are pretty much chained to boost::optional<Matrix &> for NonlinearFactors, or OptionalJacobian for many functions, as well as expressions. Is there no way to "tell" pybind11 to treat these two cases differently? And, even if a copy is made of an OptionalJacobian, maybe the Eigen::Map inside it is not copied?

[shteren1]

@dellaert using linearize is a nice alternative, but like you said it only applies to NonLinearFactor derivatives, @ProfFan demo is really nice, giving a ref to eigen matrix is cool. I understand it's not going to be an automatic wrapper solution though no? for every method that i want jacobian access to I will have to add the special python overload with the eigen ref?

[ProfFan]

I think I have a nice solution to this by adding an implicit conversion from Eigen::Ref @dellaert

See updated PR