SuperPoint + SuperGlue feature for outdoor navigation

[BioR0bot]

Can you add SuperPoint and superglue features with ncnn library for more stable navigation in changing light conditions?

[ymd-stella]

If you cannot make a more concrete proposal, I will close this.

[BioR0bot]

I'm working on a navigation project for a street robot. I noticed that the orb features map may work worse depending on the time of day, and neural networks are used for more stable tracking. I think that such tracking will work on mobile devices using the NCNN library. If you need my help, I can help with integration

[ymd-stella]

I already know the theory about it, please show me a concrete plan for stella_vslam.

[BioR0bot]

It would be nice if there was a flag to use Orb or SuperPoint, which is set at the beginning of the program execution. I can write test code for point matching and detection that uses NCNN and build it with Qt for Android to test the speed. You could then integrate the code into SLAM.

[ymd-stella]

I think the first goal should be to use SuperPoint.

Once we are sure that SuperPoint works as expected, then we can replace matcher. Additionally, what about trying LightGlue instead of SuperGlue?

[sumitsarkar1]

@ymd-stella LightGlue will be better

[jjd9]

Hello, i am also interested in the potential for using features other than orb in this library and wanted to join the discussion.

Idk if you have already considered this, but what about making the feature extraction and matching more generic as a first step on this path? I.e. rather than replacing the current implementation, it could be refactored such that any feature extractor inheriting from a standard Feature class and any matcher inheriting from a standard Matcher class could be used with stella_vslam. (This will also require revisiting the assumption made across the code-base that the features are binary and are comparable with hamming distance) This has many advantages imho

1. You have an opportunity to verify orb features still work as well as before to avoid a regression,
2. It would be easier to quantitatively compare new feature extractors and matchers to see if e.g. super point + light glue actually outperforms the current implementation of orb + brute force in the context of SLAM,
3. If someone publishes some cool new feature extractor next year that ostensibly outperforms super point, it would be way easier to test it out after this change.
4. People without a gpu could still use non-deep learning based features and matchers 😅

What do ya’ll think of this idea?

[hellovuong]

Hi guys, in this branch, I implemented SuperPoint + SuperGlue + HF_Net (global descriptor to detect loop closure beside BoW) run in TensorRT, I have not tested it on a lot of datasets but last time I checked SLAM worked fine. https://github.com/hellovuong/stella_vslam/tree/hloc

[antithing]

@hellovuong awesome! What frame rate are you seeing with that branch? Can stereo mode run at 50hz?

Have you also created the bow vocabulary to work with superpoint?

[hellovuong]

@hellovuong awesome! What frame rate are you seeing with that branch? Can stereo mode run at 50hz?

Have you also created the bow vocabulary to work with superpoint?

• It can run at 30Hz since TensorRT is fast for inference. 50Hz is a bit extreme, I am not sure if it can meet it
• Voc for SuperPoint I used from SuperPoint_SLAM, it only supports DBoW2. However, in the project that I shared you can use HF_Net as global descriptors for Loop Closure. I used naive search but it can be improved by Flann or nanoFlann KDtree to speed up the search in high dimensions.

[nnop]

So, superglue is only used in reloc? Coud is also be used in tracking? @hellovuong

[hellovuong]

So, superglue is only used in reloc? Coud is also be used in tracking? @hellovuong

SuperGlue is using for tracking.

[nnop]

Could you please provide the code position where superglue is used for tracking? I only saw it for relocalization.

[hellovuong]

Could you please provide the code position where superglue is used for tracking? I only saw it for relocalization.

The github repo doesn't up-to-date with my local change, It was a while since I work with this project. I will clean-up the code and push the changes.

[nnop]

Thanks for the responce. But before seeing your new changes, I want to ask about the tracking method you used with superglue.

Generally, tracking is a process of pose/map guided nearest neightbor feature search. So, how did you incorporate the predicted pose with superglue? or just used the pose to filter outlier matches?

[hellovuong]

@nnop I tried to match directly without the nearest neighbor feature search. The predicted pose only use for init guess of pose optimization

[nnop]

Thanks for your responce. Got it. In that way, its a sequentail SfM method instead of SLAM tracking. Still, for sequential data, the predicted pose can be used to check if the matching result is valid.