Superpoint + Tangent Images

[Artcs1]

Hi @meder411, First congrats for the excellent work, I am working with your library with the keypoint detector part I do a modification to the sift key points, to work with Superpoint but I notice when the code rendering the descriptors always return 128 dimension vector. Is there a way to change this number? In the Superpoint architecture, I have a 256 dimension vector, so I run twice the rendering with two different 128 dimension vector and then concatenating it. Does this procedure maybe affect the result? What do you think? I don't find many information in the paper for the rendering part. Is there any reference available?

Thank you @meder411

[meder411]

So SIFT is actually is baked into the implementation of the the visibile keypoints function in my spherical distortion library. (This more of a result of tight timing before the CVPR deadline so it wasn't as extensible as I'd have liked.) A good work-around is to use the gnomonic (rectilinear) projection to map the corners of the face of the icosahedron onto the corresponding tangent image and just keep the keypoints that fall within that projected triangle.

[meder411]

Take a look at this TangentUVToSpherical function in my PR to OpenMVG. This shows how to convert the coordinates from tangent image pixels back to equirectangular image pixels.

Similarly, take a look here to see how to select only pixels (or features) that fall within the valid region of the tangent image.

If I have some more time I will update this repo with that logic so any descriptor type can be used.

[meder411]

@Artcs1 Okay this was something that was bugging me for a while, so I just pushed a fixe. Version 1.1 of my spherical-distortion repo now includes a function to convert tangent image coordinates to spherical coordinates. You can see an example of how you can use this for computing SIFT descriptors in sift_tangent_images which is referenced in the updated SIFT example in this repo.

It should be fairly clear how to adapt this to SuperPoint. For example, adjusting part of the sift_tangent_images function:

# Let's say you have some keypoint info:
#     `coords` (N, 2) pixels coords on a tangent image
#     `desc` (N, whatever) descriptor you've computed (maybe SuperPoint?) at each of those coords
#  and you also have this info from the tangent image set
#     `base_order` and `sample_order` have defined your tangent image representation
#     `image_shape` is your equirectangular image dimensions
#     `tangent_img_idx` is simply the tangent image you detected on (i.e. you run this in a loop over the tangent image set)

# First, compute visible keypoints
visible_coords_ti, visibility_mask = get_valid_coordinates(base_order,
                                   sample_order,
                                   tangent_img_idx,
                                   coords,
                                   return_mask=True)
visible_desc = desc[visibility_mask]

# Convert tangent image coordinates to equirectangular
visible_coords_erp = convert_spherical_to_image(
    torch.stack(
        convert_tangent_image_coordinates_to_spherical(
            base_order, sample_order, tangent_img_idx, visible_coords_ti), -1),
    image_shape)

# Now you have the visible coordinates in equirectangular image pixel coords as `visible_coords_erp` and their corresponding descriptors as `visible_desc`.

Let me know if this helps!

[Artcs1]

Hi @meder411,

First, I adapt the code of visible keypoints function, to operate with the size of the keypoint vector, these fix my first experiment (recover pose pipeline), now the SuperPoint+TangentImages performs well under rotation on Z axis, but in Y and X axis isn't. I will test with the new function in the library and report the results.

Thank you for all!