Any plan to port this project to other popular framework?

[HTLife]

Since the project is really useful for people who work on the geometric neural network, do you have any plan to port this project to PyTorch or Tensorflow?

[ankurhanda]

I have done it in tf but releasing it some time later - not sure when. Lots of other things in the priority as of now.

[HTLife]

@ankurhanda Got it thanks!
Allow me to keep this issue open, then you could inform everyone when the porting is ready.

[ankurhanda]

okay no problem.

[HTLife]

Hi @ankurhanda, I'd like to port the "SE3 composition layer" mentioned in [1][2] to PyTorch. Could you give me some advice that what is the minima related files that I should change?

Since [1][2] do not provide any implementation detail, I found that

1. gvnn/TransformationRotationSO3.lua
2. gvnn/TransformationMatrix3x4GeneratorSO3.lua could be related.

[1] Clark, Ronald, et al. "VINet: Visual-Inertial Odometry as a Sequence-to-Sequence Learning Problem." AAAI. 2017. [2] S.Wang, R.Clark, H.Wen, andN.Trigoni, “End-to-end, sequence-to-sequence probabilistic visual odometry through deep neural networks,” Int. J. Rob. Res., 2017.

[ankurhanda]

1. is pure Rotation and 2. is rotation and translation. I don't know which one you'd need. I think you need 2.

[HTLife]

@ankurhanda Thank you! Another thing that I want to make sure is the file dependency. If I only need SE3 related function(assume it's TransformationMatrix3x4GeneratorSO3.lua), would other lua and cu files be unrelated? (TransformationMatrix3x4GeneratorSO3.lua seems to be self-contained.)

[ankurhanda]

I purposely kept them self contained so you don't need any dependencies.

[HTLife]

@ankurhanda Could you share the tf version of TransformationMatrix3x4GeneratorSO3 with me? That will be easier for me to port lua code to python with some reference. (My GitHub profile has my email address.)

[ankurhanda]

You need to implement Equation (9) here https://arxiv.org/pdf/1312.0788.pdf. I cannot release/share the code without getting internal permissions, sorry!

[HTLife]

@ankurhanda I just finished the draft of porting TransformationMatrix3x4GeneratorSO3. (code) If you have time, could you help me to check the line ended with "#????"? (I'm not confident that I port these line right.)

[ankurhanda]

I don't know much about pytorch but it seems to look all OK to me. Did you try some unit testing on it? Otherwise I'd highly recommend Equation (9) in the paper I sent you earlier. It is quite easy to implement.

[HTLife]

@ankurhanda I appreciate your kind assistance. I'll follow up the Eq9 to check the correctness of my implementation.

Beside that,

line415:  gradParams:narrow(3,1,3):narrow(3,1,3):add(1,gradientCorrection)

what is the behavior of the add function with two parameters? I only found the description about one paramter in the document of torch.

[ankurhanda]

It has been a while since I touched torch I will have to go back to it. Will come back to you later. In the meanwhile do look at Eq(9) and you don't need all of this anymore. My tf implementation didn't use any of this anymore since I found an easier way to do this using auto-diff. You only need this function https://github.com/HTLife/gvnn_pytorch/blob/master/TMSO3.py#L37 and pytorch will do the auto-diff. I'd say that you don't need backward pass layer explicitly implemented. Most of these operations come with auto-diff.

[HTLife]

@ankurhanda lol, now I get what you mean by "easy to implement".
I thought that the backprop gradient might involve some special adjustment in gvnn. If auto-diff could work, that would be way less pain to port this stuff.

[ankurhanda]

Yes, that's why I was stressing you to do it directly via auto-diff. I am happy to check your code then. The way I did it in gvnn was a slightly complicated way since torch didn't have auto-diff when I did it. Sorry for the trouble.

[HTLife]

After read through the Eq9., I found that it only contains rotation(SO3). I apologize I didn't state my goal clearly.

My goal is to implement an NN layer that can combine Global pose (SE3, denoted with Tg) and related pose(se3, denoted with Xi).

Therefore, I only need to implement the exponential mapping from se3 to SE3 and the Matrix multiplication in forward pass. Then auto-grad should do the backward for me.

def expo(xi):
  ...

def forward(self, Tg, xi):
  Txi = expo(xi)
  return torch.mul(Tg, Txi)

Did I understand the whole process correctly?

[ankurhanda]

So, in summary, you need to implement exponential map both for SO3 and SE3. You should implement Eq (15) and Eq (84) here http://ethaneade.com/lie.pdf. If you google SO3/SE3 you may find other tutorials on them that you may like but this pdf is my favourite.

Also, make sure to check the norm of se3 and so3 and so if the norm is < 1e-8 etc. switch to linear approximations, otherwise you will see NANs.

[HTLife]

Thanks for your kind explanation!

[ankurhanda]

happy to see your code once you have written it!

[HTLife]

Yes, I'll open-source my implementation.

One more thing to ask: As your experience, will the auto-grad taking care of the hat operation*?

Auto-grad seems to be reasonable to handle multiplication and addition in forward pass. However, hat operation is about assigning vector elements to a certain position of the matrix.

• hat operation: convert 6-vector representation of se3 to 4x4 Matrix representation or convert the 3-vector representation of so3 to 3x3 matrix representation

[ankurhanda]

which equation are you talking about? hat operator?

[HTLife]

[ankurhanda]

You need the matrix isn't it? that's how you will get 3x3 rotation matrix. Look at Equation (15) again. You just need to implement that. The skew symmetric conversion from (2) to (3) should be easy to do in pytorch using pytorch operations. They will allow auto-diff.

[ankurhanda]

Eq(15) is what I implemented here (but in torch) https://github.com/ankurhanda/gvnn/blob/master/TransformationRotationSO3.lua#L254-L304. You just need to exactly that in pytorch. Are you not doing that?

[HTLife]

Yes, I'm doing Eq(15) and Eq(84).

In forward(Tg, xi), xi need to be represented in 6-vector form. Therefore, before I do exp map, I need to convert 6-vector to R^4x4 matrix. The conversion is easy, I only need to fill in matrix according to the hat-opration definition.

However, my concern is how can auto-diff know how to reverse process of hat-operator, and do the backprop?

[ankurhanda]

You do not need to convert the 6-vector to 4x4 before the exp. That is what I have been trying to stress. Look again the code I sent you in the previous message. I extract elements 0, 1, 2 (for SO3) and turn them into skew symmetric matrix and then compute sin, 1-cos and then add them together. All operations support auto-diff. Th conversion from 3-vec to 3x3 skew symmetic matrix is all you need to do first. Look at my torch code again - I'm sure such conversion can be done in pytorch as well.

[HTLife]

@ankurhanda I finished the forward pass implementation and checked with some test case. So far so good. ^_^ How do you get your test case for the backward pass?

[ankurhanda]

if you use auto-diff you don't need backward pass checks - only forward pass sanity checks. You should also ensure that (0, 0, 0, 0, 0, 0) should give you [ Identity(3) | zeros(3,1)] matrix for SE3. I didn't see anywhere you are using linearisation when the norm is small.

[HTLife]

Thanks for advices! I fixed these issue and related code is uploaded.

1. The all-zero condition of Tg and xi could be handle correctly.
2. sin(theta)/theta seems don't have numerical problem when theta is small

   theta = torch.from_numpy(np.array([1e-1, 1e-2, 1e-3, 1e-4, 1e-5, 1e-6, 1e-7, 1e-8, 1e-9, 1e-10]))
   one = torch.sin(theta) / theta
   print(one)
   """
   tensor([ 0.9983,  1.0000,  1.0000,  1.0000,  1.0000,  1.0000,  1.0000,
        1.0000,  1.0000,  1.0000], dtype=torch.float64)
   """         

3. For (1.0 - torch.cos(theta)) / (theta*theta)and (theta - torch.sin(theta)) / (theta**3), theta greater than 1e-1 will be calculte directly. 1e-1 to 1e-6 will be calculate by Taylor expansion. Limitation value will be assigned when theta smaller than 1e-6. (These threshold is been defined by observing the value difference between close form solution and Taylor expansion.)

(1.0 - torch.cos(theta)) / (theta*theta)

import numpy as np
np.set_printoptions(precision=15)
torch.set_printoptions(precision=15)
theta = torch.from_numpy(np.array([1e-1, 1e-2, 1e-3, 1e-4, 1e-5, 1e-6, 1e-7, 1e-8, 1e-9, 1e-10]))

co = (1.0 - torch.cos(theta)) / (theta*theta)
print('by formula')
print(co)

co_l = torch.full((theta.shape[0],), 1.0 / 2.0, dtype=torch.float64)
co_l += torch.pow(theta, (4*1)) / 720.0# 6!
co_l += theta**(4*2) / 3628800.0# 10!
co_l -= theta**(2) / 24.0
co_l -= theta**(2 + 4) / 40320.0

print('by Taylor')
print(co_l)

by formula
tensor([0.499583472197418, 0.499995833347366, 0.499999958325503, 0.499999996961265, 0.500000041370185, 0.500044450291171, 0.499600361081320,
        0.000000000000000, 0.000000000000000, 0.000000000000000], dtype=torch.float64)
by Taylor
tensor([0.499583472197423, 0.499995833347222, 0.499999958333335, 0.499999999583333, 0.499999999995833, 0.499999999999958, 0.500000000000000,
        0.500000000000000, 0.500000000000000, 0.500000000000000], dtype=torch.float64)

(theta - torch.sin(theta)) / (theta**3)

si = (theta - torch.sin(theta)) / (theta**3)
print(si)

si_l = torch.full((theta.shape[0],), 1.0 / 6.0, dtype=torch.float64)
si_l += theta**(4*1) / 5040.0
si_l += theta**(4*2) / 39916800.0
si_l -= theta**(2) / 120.0
si_l -= theta**(2 + 4) / 362880.0
print(si_l)

tensor([0.166583353171851, 0.166665833335744, 0.166666658339004, 0.166666661483190, 0.166667284899440, 0.166653732372284, 0.172053567411030,
        0.000000000000000, 0.000000000000000, 0.000000000000000], dtype=torch.float64)
tensor([0.166583353171848, 0.166665833335317, 0.166666658333334, 0.166666666583333, 0.166666666665833, 0.166666666666658, 0.166666666666667,
        0.166666666666667, 0.166666666666667, 0.166666666666667], dtype=torch.float64)