search

rpng / open_vins

An open source platform for visual-inertial navigation research.
https://docs.openvins.com
GNU General Public License v3.0
2.06k stars 616 forks source link

Implementing custom feature tracking and description #343

Closed zakih closed 12 months ago

zakih commented 1 year ago

Motivation: I am interested in trying out different feature tracking/matching and perhaps descriptors as well. Looking for advice from someone with a solid understanding of the codebase.

Current understanding: I see that /ov_core/src/track/ and /ov_core/src/feat/ are the place to do that

Questions:

  1. Is it possible/feasible to implement custom feature trackers and/or descriptors?
  2. How exactly can I achieve this with the way the code is currently setup?
  3. What are the minimal interfaces/dependencies I should be mindful of (e.g. the feature database)?

Thank you

goldbattle commented 1 year ago

Yes! All tracking is self contained in the /ov_core/src/track/. There is some very naive descriptor implementation in TrackDescriptor. This is probably the first place to start. We mostly use the KLT, so be prepare for the descriptor code to not work as well as it isn't well tested on all the datasets. The current descriptor used is ORB.

Basically, all what you need to provide is the uv pixel location, and a unique ID over time. How you get this from an inputted image is up to you.

https://github.com/rpng/open_vins/blob/58afb2daf47498911100edc7452ccd7e42a2db50/ov_core/src/track/TrackDescriptor.cpp#L151-L155

zakih commented 11 months ago

Thanks @goldbattle one more question: is it possible to use Open-VINS without a feature description component? I have a subroutine that simply outputs matching pixel pairs across two images and no descriptors of each of the pixels in the two images. Note, this is different than the typical two-step method where features are extracted and then matched. I realize, one complication may arise when we try to do data association (loop closures) without having a database of feature descriptors. But I'm hoping to run Open-VINS without a feature descriptor while sacrificing any scene matching functionality. Is this doable?

goldbattle commented 11 months ago

No loop-closure is currently performed so I think you should be all good. I would refer to you the TrackKLT which does sequential tracking. You might just be able to directly replace the perform detection and matching functions with what you have. https://github.com/rpng/open_vins/blob/9d499dbc2f87bb8ef115e7cf40ac39f6376202ca/ov_core/src/track/TrackKLT.cpp#L125-L140

You don't need detect as long as you have some logic to handle the first ever frame / first observed feature. Hope this helps!