Support for open version of Superpoint

[adricostas]

Hello,

I have just found out that a new repository to train Lightglue is available (https://github.com/cvg/glue-factory). They have included the open version of Superpoint this time so I was wondering if you are planning to include this also into LightGlue-ONNX.

Thank you very much in advance!

[fabio-sim]

Hi @adricostas

Thank you for the information. I am able to export SuperPoint-Open, however, it seems that when I try to pass its output features to the LightGlue matcher that was trained on SuperPoint-MagicLeap's features, I obtain a blank output (zero matches).

I cannot find any weights for a LightGlue that has been trained on SuperPoint-Open's features. Could you point me to a URL for these weights?

[adricostas]

Hi,

Well, I'm not sure, but they say that: "All models and datasets in gluefactory have auto-downloaders, so you can get started right away!". So, inside the code of the model of each extractor/matcher exists an url from which the weights are downloaded. However, as you pointed out, it seems that in the LightGlue case, only the weights for the not open version of SuperPoint are available: https://github.com/cvg/LightGlue/releases. I guess that it is necessary to train the LightGlue network by yourself using the outputs of the open version of SuperPoint. I don't know if this makes sense.

Thanks

[fabio-sim]

@adricostas It appears that LightGlue weights trained on SuperPoint-Open haven't been released yet. See https://github.com/cvg/glue-factory/issues/22

[adricostas]

Ok, thanks for the information! Let's wait for them, then.

[thejasonfisher]

From: https://github.com/rpautrat/SuperPoint

Update: We have converted the Tensorflow model into a Pytorch one. This provides a version of SuperPoint with MIT license, which can be used with the Pytorch model defined in superpoint_pytorch.py.

It sounds like those weights might work? https://github.com/rpautrat/SuperPoint/blob/master/weights/superpoint_v6_from_tf.pth

[noahzn]

Hi @adricostas

Thank you for the information. I am able to export SuperPoint-Open, however, it seems that when I try to pass its output features to the LightGlue matcher that was trained on SuperPoint-MagicLeap's features, I obtain a blank output (zero matches).

I cannot find any weights for a LightGlue that has been trained on SuperPoint-Open's features. Could you point me to a URL for these weights?

HI, can you share your code to export superpoint-open? I tried by myself, but I have warnings like this UserWarning: Exporting aten::index operator of advanced indexing in opset 17 is achieved by combination of multiple ONNX operators, including Reshape, Transpose, Concat, and Gather. If indices include negative values, the exported graph will produce incorrect results. warnings.warn(

UserWarning: The exported ONNX model failed ONNX shape inference. The model will not be executable by the ONNX Runtime. If this is unintended and you believe there is a bug, please report an issue at https://github.com/pytorch/pytorch/issues. Error reported by strict ONNX shape inference: [ShapeInferenceError] Inference error(s): (op_type:MaxPool, node name: /MaxPool): [ShapeInferenceError] Attribute dilations has incorrect size (Triggered internally at ../torch/csrc/jit/serialization/export.cpp:1484.) _C._check_onnx_proto(proto)

[fabio-sim]

Hi @noahzn, thank you for your interest in LightGlue-ONNX.

This is the code I adapted when exporting SuperPoint-Open: https://github.com/fabio-sim/LightGlue-ONNX/blob/fc1d67a361ec52405d792987f019f4c047549bc3/lightglue_onnx/superpoint_open.py#L1-L188

As for the error/warning, a lot has changed with respect to the frameworks though, so you might have to align to these versions iirc:

torch==2.1.0+cu118
onnx==1.14.1
opset 15/16

Hope this helps!

[noahzn]

@fabio-sim Thanks a lot!!