An end-to-end approach for model transfer. Use thin-plate splines to estimate the optical flow and then use the multi-scale in-painting model to produce the final results.
Background motion predictor predict the background affine transform($E_{bg}$)
Keypoint detector detect KxN key points for each image. Create TPS transform from every N key points ($E_{kp}$)
Dense Motion Block predict the optical flow and multi-scale occlusion mask from above-mentioned K+1 transform. Specifically, it predicts the weights of each transform and we can combine the transforms by softmax function
Inpaint Network encode the source image and apply occlusion mask to every wrapped feature layers. the decoder part then produce the final results
Loss function
Perceptual loss
Equivariance loss to constraint the key points: $L{eq} = | E{kp}(\tau{ran}(S)) - \tau{ran}(E_{kp}(S) |$ for FOMM[^1] and MRAA[^2]
Inpainting loss: $L_{warp} = \sum | \tau(E_i(S)) - E_i(D) |$, $E_i$ is the i-th layer of inpainting network
Highlight
Dropout for TPS transformers: In training, randomly turn off one of the K TPS transform and assign the equal weight to the rest in order to prevent strapped in the local minimum
The animation via disentanglement(AVD) model to apply the relative motion to better animate the target if the shape of source and target are not match. see MRAA[^2]
Limitation
Failed in the extreme cases, for example dog and giraffe
Introduction
An end-to-end approach for model transfer. Use thin-plate splines to estimate the optical flow and then use the multi-scale in-painting model to produce the final results.
Github
Method
Loss function
Highlight
Limitation
Comments
[^1]: [FOMM] First order motion model for image animation [^2]: [MRAA] Motion representations for articulated animation - https://snap-research.github.io/articulated-animation