search

naver / kapture

kapture is a file format as well as a set of tools for manipulating datasets, and in particular Visual Localization and Structure from Motion data.
BSD 3-Clause "New" or "Revised" License
475 stars 66 forks source link

Multiple types of keypoints and descriptors #8

Closed ducha-aiki closed 3 years ago

ducha-aiki commented 3 years ago

Hi,

I am trying to fit the kapture for my use case, which requires multiple types of local features, both detectors and descriptors, e.g. SuperPoint, R2D2, SIFT+HardNet. The current kapture format assumes a single type of local feature -- which is hardcoded in the filenames "keypoints.txt", "descriptors.txt". If I would make a fork for support multiple feature types, would it be merged at some point, or do you not plan to change the format?

Best, Dmytro.

humenbergerm commented 3 years ago

Hi Dmytro!

Thanks for your interest in this. Indeed, we already discussed a lot how to support multiple types of local features. We will continue the discussion and get back to you as soon as possible.

Best, Martin

ducha-aiki commented 3 years ago

Thank you, Martin! If possible, I would be happy to join the discussion :) My email is ducha.aiki@gmail.com just in case

humenbergerm commented 3 years ago

Great! It would also be helpful if you could share some more details about your needs. At the moment we use kapture with multiple features, but not within the same reconstruction. Best, Martin

ducha-aiki commented 3 years ago

Sure. The general idea is to revisit MODS, which combines the multiple type of features in order to speed-up matching of the easy pairs and to allow matching the hard pairs, where any single feature fails, but combined they work. So, first we are matching everything which is matchable by ORB, then we add, say, SuperPoint, and so on.

https://github.com/ducha-aiki/mods-light-zmq https://arxiv.org/abs/1503.02619

yocabon commented 3 years ago

Hi, We've started to draft a new version of the format (1.1) that should hopefully answer your needs. The new specifications are being written here https://github.com/naver/kapture/blob/1.1_draft/kapture_format.adoc

There are no changes to the "sensor" part of the format. In the "reconstruction" part, multiple keypoints/descriptors/global features types can be given. keypoints can be represented by multiple descriptors. Observations can reference different types of keypoints.

What do you think of these changes ?

ducha-aiki commented 3 years ago

Hi,

It fits my use-case, thank you! One thing, I would like to add is a metric for descriptors. So the descriptors.txt would be:

name, dtype, dsize, keypoints_type, metric_type

where the metric is "L2" for the most and "Hamming" for the binary features, for example.