Question: Compatibility between `sym` and `symforce` types

[joansola]

I started using symforce short ago, and I am also a newbie in python. But here goes my question:

I have set an optimization problem by taking the example in the main README and adapting it to my needs.

The data types there are sf.Pose3 and sf.V3 mainly, and some combinations.

I sometimes want to specify numeric types, and so I started using sym.Pose3.

To my surprise, I encounter this anomaly:

• position is sym.Pose3.position() but sf.Pose3.t.
• orientation is sym.Pose3.rotation() but sf.Pose3.R

I can't make code that accepts both. For instance, I have a residual function f(a,b) where a is numeric and b is symbolic. If I call it as f(c,d) where the types of c,d differ from those of a,b, I encounter the conflics between .t and .position(), in both directions.

I would like to make sense of all this. Should I use exclusively types from symforce? If so, how to indicate that they are numeric?

[aaron-skydio]

Generally yes, typically we've been defining functions that use types from symforce. You can do numerical computations with these types as well, not just symbolic ones - for instance, if you create a Pose with pose = sf.Pose3.symbolic('P'), and look at pose.to_storage(), you get:

[P.R_x, P.R_y, P.R_z, P.R_w, P.t0, P.t1, P.t2]

But you could similarly create an identity pose sf.Pose3.identity(): [0, 0, 0, 1, 0, 0, 0]

Or some other numerical pose sf.Pose3(R=sf.Rot3.from_angle_axis(angle=0.3, axis=sf.V3(0, 0, 1)), t=sf.V3(0.1, 0.2, 0.3)): [0.0, 0.0, 0.149438132473599, 0.988771077936042, 0.1, 0.2, 0.3]

Doing numerical operations on symforce types isn't particularly fast though. If you want a faster function that operates on the numeric sym types, we recommend generating the numerical python function from the symbolic one, as described in the Codegen tutorial. You can also immediately load the generated function and call it with the sym types, as you can see that the bottom of that tutorial.

That being said, we do want the APIs of the sym and symforce types to be as similar as possible. Fundamentally, the internal storage of the two types is different, which is why we have those different accessors (the sym types store a flat vector of the necessary scalars, while the symforce types store the rotation and position components as individual objects). But I think it'd be reasonable to either add t and R properties to the generated pose types, or add position() and rotation() functions to the symbolic types so there's a consistent way to access those things

[joansola]

Hi Aaron thanks for this detailed explanation!

By now I'm just typing some small mock-ups so performance is not an issue.

I however ran into a number of oddities (and that's why I started trying with sym objects, to try to circumvent them). Let me comment some of them here FYI. If these are really issues in symforce, I can open issues for them. But as I am badly skilled in Python I'd prefer to just comment them to you here.

1. Constructing a list of robot poses incrementally:

   This works:

   robot = sf.Pose3.identity()
   v_dt = np.array([0,0,0, 1,0,0])
   robots = []
   
   for t in range(3):
      robots.append(robot)
      robot = robot.compose(sf.Pose3.from_tangent(v_dt, sf.numeric_epsilon))
   
   display(robots)
   
   >>>
   [<Pose3 R=<Rot3 <Q xyzw=[0, 0, 0, 1]>>, t=(0, 0, 0)>,
   <Pose3 R=<Rot3 <Q xyzw=[0, 0, 0, 1.0]>>, t=(1, 0, 0)>,
   <Pose3 R=<Rot3 <Q xyzw=[0, 0, 0, 1.0]>>, t=(2, 0, 0)>]

   This does not work: <-- all robots in the sequence get the same value!

   robot = sf.Pose3.identity()
   v_dt = np.array([0,0,0, 1,0,0])
   robots = []
   
   for t in range(3):
      robots.append(robot)
      robot.t[0] = robot.t[0] + 1
   
   display(robots)
   
   >>>
   [<Pose3 R=<Rot3 <Q xyzw=[0, 0, 0, 1]>>, t=(3, 0, 0)>,
   <Pose3 R=<Rot3 <Q xyzw=[0, 0, 0, 1]>>, t=(3, 0, 0)>,
   <Pose3 R=<Rot3 <Q xyzw=[0, 0, 0, 1]>>, t=(3, 0, 0)>]

   This is the reason I started using numeric types because I thought the issue resided in the sf type.

2. sf.Pose3 is not copyable To solve the issue above, I tried to copy and deepcopy the robot object above, without success:

   import copy
   robot = copy.deepcopy(robot)  # <--- fails

   the error message is

     Output exceeds the [size limit](command:workbench.action.openSettings?[). Open the full output data [in a text editor](command:workbench.action.openLargeOutput?8445dd96-f21c-4ec6-863a-d44dad16ee96)
   ---------------------------------------------------------------------------
   TypeError                                 Traceback (most recent call last)
   /Users/jsola/dev/allaus/python/examples/drafts.ipynb Cell 6 in <cell line: 3>()
        [1](vscode-notebook-cell:/Users/jsola/dev/allaus/python/examples/drafts.ipynb#W5sZmlsZQ%3D%3D?line=0) import copy
   ----> [3](vscode-notebook-cell:/Users/jsola/dev/allaus/python/examples/drafts.ipynb#W5sZmlsZQ%3D%3D?line=2) robot  = copy.deepcopy(robot)
   
   File ~/mambaforge/envs/symforce/lib/python3.10/copy.py:172, in deepcopy(x, memo, _nil)
      170                 y = x
      171             else:
   --> 172                 y = _reconstruct(x, memo, *rv)
      174 # If is its own copy, don't memoize.
      175 if y is not x:
   
   File ~/mambaforge/envs/symforce/lib/python3.10/copy.py:271, in _reconstruct(x, memo, func, args, state, listiter, dictiter, deepcopy)
      269 if state is not None:
      270     if deep:
   --> 271         state = deepcopy(state, memo)
      272     if hasattr(y, '__setstate__'):
      273         y.__setstate__(state)
   
   File ~/mambaforge/envs/symforce/lib/python3.10/copy.py:146, in deepcopy(x, memo, _nil)
      144 copier = _deepcopy_dispatch.get(cls)
      145 if copier is not None:
   --> 146     y = copier(x, memo)
      147 else:
   ...
      163     reductor = getattr(x, "__reduce__", None)
   
   File stringsource:2, in symengine.lib.symengine_wrapper.MutableDenseMatrix.__reduce_cython__()
   
   TypeError: no default __reduce__ due to non-trivial __cinit__

[aaron-skydio]

1. This is a standard python gotcha, not particular to symforce - appending to a list does not make a copy of the object you append (all Python objects are pointers under the hood), so you're creating a list where each entry points to the same pose object, and each time you execute robot.t[0] = robot.t[0] + 1 you're modifying t on that one object

2. Indeed deepcopy might be one solution to the above :) I'm not particularly shocked that deepcopy does not work on symengine objects. You should be able to effectively achieve a deep copy by doing robot = sf.Pose3.from_storage(robot.to_storage()). This is really a symengine issue/bug, but I've made issue #219 to track.

[joansola]

Thanks again Aaron. your comments are really helpful.

So I am getting familiar with all this, and I can now make code that does what I want.

I want to thank you for FINALLY getting an optimizer out there that works with automatic Lie derivatives!

[aaron-skydio]

Thanks for trying it out! Definitely feel free to make more issues if/when you run into more things :)