Nonrigid deformation not working for 3D to 2D projection

[gaetanmrnd]

Hi, Thanks for your CPD implementation.

I'm having trouble matching a 3D model to points extracted from a 2D picture. It works fine with the affine deformation, but the result is not very precise (I'm guessing because the pinhole camera model needs nonrigid deformation).

When I try the nonrigid deformation, the source data is not modified and the error is unbelievably small. I don't understand, am I doing anything wrong or is there a problem with the deformable_registration class? I'm attaching my code (very close to the example code) and the results in both cases.

Thanks,

Gaetan

from functools import partial
from mpl_toolkits.mplot3d import Axes3D
from pycpd import deformable_registration, affine_registration
import time

def visualize(iteration, error, X, Y, ax):
    plt.cla()
    ax.scatter(X[:,0],  X[:,1], X[:,2], color='red', label='Target')
    ax.scatter(Y[:,0],  Y[:,1], Y[:,2], color='blue', label='Source')
    ax.text2D(0.87, 0.92, 'Iteration: {:d}\nError: {:06.4f}'.format(iteration, error), horizontalalignment='center', verticalalignment='center', transform=ax.transAxes, fontsize='x-large')
    ax.legend(loc='upper left', fontsize='x-large')
    plt.draw()
    plt.pause(0.001)

target = np.loadtxt("PointClouds/" + testp[:-4] + ".pcd", skiprows = 11)
X1 = np.zeros((target.shape[0], target.shape[1] + 1))
X1[:,:-1] = target
X2 = np.ones((target.shape[0], target.shape[1] + 1))
X2[:,:-1] = target
X = np.vstack((X1, X2))

source = np.loadtxt('SmallFrame.pcd', skiprows = 11)
Y1 = np.zeros((source.shape[0], source.shape[1] + 1))
Y1[:,:-1] = source
Y2 = np.ones((source.shape[0], source.shape[1] + 1))
Y2[:,:-1] = source
Y = np.vstack((Y1, Y2))

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
callback = partial(visualize, ax=ax)

reg = deformable_registration(**{ 'X': X, 'Y': Y })
reg.register(callback)
plt.show()

Affine:

Nonrigid:

[siavashk]

You can set alpha to a smaller value. This will reduce the regularization force that keeps the point together during non-rigid transformation.

With that being said, non-rigid deformation is not a suitable transformation for modelling a 3D-to-2D projection. You should look into a projection matrix. Unfortunately, I have not implemented such a transformation, and I do not intend to as this was not covered in the original CPD paper.