Transfer coordinate from Kinect1 to Kinect0

I am trying to transfer the 15th joint from Kinect1 (behave\sequences\Date01_Sub01_backpack_back\t0005.000\k1.color.json) to Kinect0 (behave\sequences\Date01_Sub01_backpack_back\t0005.000\k0.color.json), because 15th joint isn't annotated correctly for Kinect0 in this particular example.

To begin with, what I am doing is, taking the Kinect1 15th joint XY coordinates from Date01_Sub01_backpack_back\t0005.000\k1.color.json, do back-projection to get camera-relative coordinates. Multiply with depth to get 3D coordinates.

image_coordinates = body_joints # [1072.214111328125, 586.692626953125] 
homogeneous_coordinates = [image_coordinates[0], image_coordinates[1], 1] 
# [1072.214111328125, 586.692626953125, 1]

homogeneous_coordinates = np.atleast_2d(homogeneous_coordinates).T
# [[1072.214111328125], 
# [586.692626953125],
# [1]]

_, inv_K = load_intrinsics(camera="1")
# [[0.0010206326776383019,             0.0,                         -1.0399760465284908],
# [0.0,                                   0.0010205747597485165,   -0.7955244457990647]
# [0.0,                                                0.0,                                          1.0               ]

inv_K_homogeneous_coordinates = np.matmul(inv_K, homogeneous_coordinates)
# [[0.05436071291790556],
# [-0.1967607590001531],
# [1.0]]

_3D_camera_relative_coordinates = inv_K_homogeneous_coordinates * depth
# [[133.78171449096558],
# [-484.2282278993768],
# [2461.0]]

Now transfer 3D coordinate to Kinect0

homogeneous_camera_relative_coordinates = np.append(_3D_camera_relative_coordinates, [[1]], axis=0)
# [[133.78171449096558],
# [-484.2282278993768],
# [2461.0],
# [1]]

# inverse of Kinect0 extrinsics
transformation_matrix = [[0.21706500345078444,-0.004850319291858198,0.9761451012424724,-2.4012728998616777],
[0.0011704507569507935,0.9999882298750086,0.004708519562866193,-0.043870251217095806]
[-0.9761564497158927,0.0001204749574813712,0.21706812562844988,1.7749694834160328]]

transformed = np.matmul(transformation_matrix, homogeneous_camera_relative_coordinates)
# [[2431.27981109], [-472.52214717], [ 405.32940583]]

K, _ = load_intrinsics(camera="0")
# [[976.2120971679688, 0, 1017.9580078125], [0, 976.0467529296875, 787.3128662109375], [0, 0, 1]]

temp = np.matmul(K, transformed)
# [[ 2.78605308e+06], [-1.42082651e+05], [ 4.05329406e+02]]

x = temp[0][0] / temp[2][0] # 6873.552813113347
y = temp[1][0] / temp[2][0] # -350.5362530603442

These x, and y values are wrong, because the 15th coordinate in Kinect0 should be within the image

xiexh20 / behave-dataset

Transfer coordinate from Kinect1 to Kinect0 #6