EthanRosenthal
commented
2 years ago If I understand your question correctly, you only have a set of keypoints for each image, but you don't know if keypoint 1 for image A is referencing the same location as key point 2 for image B?
If that's the case, I think that you can still do the image registration, but you will have to look out for one thing. In my code, I create a Gaussian mask around each keypoint. Each keypoint gets its own index in the torch tensor. I know that the keypoint at index 2 in image A corresponds to the same location of the body as the keypoint in index 2 in image B. I then create Gaussian masks centered at each keypoint, multiply these masks by the body "segmentation mask", and then perform image registration on the resulting mask. Crucially, that resulting mask has an index for each keypoint.
In your case, you could take the sum along the keypoint index. You would end up with a single mask that contains all of the different keypoint Gaussian masks and then perform image registration on this resulting mask. While this can work, the danger could be that the image registration algorithm ends up registering the wrong keypoints on each other if that happens to provides the optimal solution. You won't know if that's an issue until you try it, though!
Thank you for sharing such an amazing work of yours!
I'm studying image registration and always wondered how I can replace optimization part with PyTorch.
Your work is exactly what I've been looking for.
I have a question regarding a mapping process.
Using segmentation model, you obtain keypoints and use them for mapping.
What if keypoint names, such as left ear, are not provided?
Let's say we're using all keypoints from the segmentation and trying to register the image.
Does the algorithm work fine or is there any other approach?
(I'm currently working on a case in which no keypoint name information is given, but only arbitrary keypoints are given.)