Import external keypoints and descriptors?

[Phillip-Thomas]

Hi!

I am hoping to find a way to import external keypoints and descriptors into speedyvision for use in a forward-backward Lucas Kanade pipeline. I don't see any mention of external import in the documentation so I figured I would reach out.

My current project which intends to accomplish world tracking operates as follows:

1) generate speedyvision keypoints and descriptors 2) add keypoints to shared memory with wasm which handles mapping 3) use opencv in WASM side to perform forward-backward tracking between keyframe map points and update their locations.

Instead of utilizing opencv in my wasm, I am trying to find a way to perform all mapping functionality and point cloud storage in the WASM but then update map point shared memory, gather the simple data structure on javascript side, and utilize it in speedyvision tracking pipeling.

Is this possible?

[alemart]

Hi! Use Speedy.Keypoint.Source to import keypoints into a pipeline. The present implementation does not support importing descriptors, but they may be recomputed easily if you provide an image. In addition, descriptors are not necessary for the Speedy Vision LK tracker.

[Phillip-Thomas]

Thank you Alexandre, I have been able to implement the external keypoint import into a pipeline, but I have one more question...

Within the keypoint object array I am importing, each keypoint has some crucial data along with it such as an ID as recognized by my 3D map and a boolean which tells if it is a projected 3d keypoint or a simple 2d keypoint that has not yet been included in a 3D map.

Currently, any excess data on the imported object that is not the 2D position values is lost along the way. Is there a way I can preserve this data during the import process so that the resulting tracked keypoints still have record of their previous id's before filtering?

[alemart]

You cannot upload arbitrary data to the GPU with a keypoint source, but you can design your pipeline differently. Perhaps you can match the keypoints and use the matching results to indirectly get the IDs and the data.

[Phillip-Thomas]

Thank you for all the help! I hope you don't mind all the questions as I have one more... Is there a way to use a distance filter to set a minimum rather than a maximum distance of keypoints during extraction?

For more context, my goal is to get keypoints that are evenly distributed across the video frame. The difficulty in doing so comes from the need for limitation of keypoints to an amount viable in a real-time 6dof slam engine. If I get more keypoints then the capacity allows, it seems to select those in order of pixel location from top-left to bottom-right.

As can be demonstrated in the image below, which is just looking at a black screen, this can often lead to all keypoints coming from the top portion of the camera stream and completely neglecting the bottom portion.

We can mitigate this slightly by trying to balance clipper size and detector threshold, but this can become quite complex and difficult to predict as the cameras view changes and lighting differs.

We can also get more keypoints than needed and iteratively remove those with locations too close but this will lead to far too much computational overhead.

I have also tried to experiment with using the shuffler, but as I understand it, doing so inherently leads to getting keypoints with lower scores, and we run into the same problem regardless, just to a lesser degree.

[alemart]

Use a Speedy.Keypoint.Clipper to get the keypoints with the highest scores. Typically, a keypoint score is a cornerness measure, but it need not be so. A score is just a number, and you can upload arbitrary scores to the GPU. If you need the keypoints to be approximately evenly distributed spatially, you can make a score function s(x,y) based on the position of the keypoint in the image, with peaks on a grid.

[Phillip-Thomas]

Is there a way to include a custom function within a single pipeline? Or would I have to have two seperate pipelines, one for extraction, and one for clipping, then run an external scoring function between?

[alemart]

You may use a keypoint source if it fits your needs. If you would like to generate the scores from within a pipeline, then a new node needs to be developed. The latter is a faster choice for your use case.

Or would I have to have two seperate pipelines, one for extraction, and one for clipping, then run an external scoring function between?

Not a good idea, because there is performance loss when moving the data back and forth to and from the GPU like that. It would be better to clip on the CPU in this case. It would be even better to get it all clipped and done on the GPU at once.