search

symforce-org / symforce

Fast symbolic computation, code generation, and nonlinear optimization for robotics
https://symforce.org
Apache License 2.0
1.41k stars 145 forks source link

Image Jacobians #351

Closed BenedictChannn closed 1 year ago

BenedictChannn commented 1 year ago

Hi all,

If my residual is a pixel of a CV image (indexed sf.DataBuffer), the Jacobians and Hessians are set to zero. I need to account for the gradient of the CV image and also the Jacobian of the reprojection of a 3D point from a reference frame to the current frame. Defining a custom Jacobian would not be ideal since I am dealing with a lot of symbolic variables. Is there a way to deal with this? Especially since the CV image is now represented as a 1D array instead the standard Matrix.

image Ideally, the autogenerated Jacobians should include the image intensity gradient and the Jacobian of the warping function as well.

Also, is there a way to handle conditional statements for symbolic computations? Are there other similar functions/ capabilities like those in logic.py

I also noticed that when using sf.logical_or operations with more than 2 arguments (unsafe = True) ,output is max(True, False) when it should by right be True?

I would appreciate any help on the above!! Thanks.

aaron-skydio commented 1 year ago

Addressed here, will continue discussion there: https://github.com/symforce-org/symforce/issues/348#issuecomment-1631701849

In particular, this is not a bug - a DataBuffer is a piecewise constant function of its index, and if you want a notion of an image gradient you need to represent that