ASpanFormer (LofTR-like) support

[fatihkykc]

Hi, would it be possible to create an example for ASpanFormer model, a loftr like model, that is mostly using the same code as loftr. I tried to implement it using your loftr example, but first, got an error from onnx saying that the opset 12 doesn't support grid_sampler. error below: Exception has occurred: UnsupportedOperatorError Exporting the operator 'aten::grid_sampler' to ONNX opset version 12 is not supported. Support for this operator was added in version 16, try exporting with this version. File "convert_to_onnx_aspan.py", line 35, in main torch.onnx.export( File "convert_to_onnx_aspan.py", line 70, in <module> main() After setting the opset version to 16, the code successfully exported the onnx model, but when calling the inferenceSession as sess = onnxruntime.InferenceSession("aspanformer.onnx") got another error saying: Exception has occurred: InvalidGraph [ONNXRuntimeError] : 10 : INVALID_GRAPH : Load model from aspanformer.onnx failed:This is an invalid model. Type Error: Type 'tensor(bool)' of input parameter (/coarse_matching/And_1_output_0) of operator (ReduceMax) in node (/coarse_matching/ReduceMax_2) is invalid.

The exported model even passes the onnx.checker.check_model validation.

Any help would be great, and thanks for the good work.

[xmba15]

Hi. Here is my quick guess: You need to convert bool type somewhere to int, somehow like this: https://github.com/xmba15/LoFTR/commit/6cdf54498c3340dee0cc02d9406b60b2a0ecd541

[xmba15]

Still WIP but if you intend to use new onnxruntime version. You might need to take a look at this. https://github.com/xmba15/onnx_runtime_cpp/commit/567a7b9bc2429154622e3b8f5bb201e058eec092

[xmba15]

If it is possible, I am more than happy if you can create a PR to integrate your work on ASpanFormer into this repo.

[fatihkykc]

Hi @xmba15, sorry for the late reply, it is possible for me to integrate the work for ASpanFormer into this repo after resolving the problems. Modifying the type like in the commit you provided resolved the problem, but another error occured on inference, error: RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Add node. Name:'/fine_matching/Add_2' Status Message: [D:\a\_work\1\s\onnxruntime\core/providers/cpu/math/element_wise_ops.h:560](file:///D:/a/_work/1/s/onnxruntime/core/providers/cpu/math/element_wise_ops.h:560) onnxruntime::BroadcastIterator::Append axis == 1 || axis == largest was false. Attempting to broadcast an axis by a dimension other than 1. 4 by 9

I believe it is caused by this line, which occurs in the loftr code too: mkpts1_f = data['mkpts1_c'] + (coords_normed * (W // 2) * scale1)[:len(data['mconf'])]

in fine_matching.py, function get_fine_match(). I will look into it now, if you guys have any advice or experience considering this error, help would be great, if not, i will comment after resolving it

PS: I am trying to use the outdoor version of aspanformer, since in my experience the indoor model doesn't perform as good as the outdoor and is preferable in my use case.

[fatihkykc]

worked around the error by removing the slicing part [:len(data['mconf'])] not tested enough to know if it effects performance or if the tensor shape mismatches in some cases. I can export the outdoor loftr model too, using the same modification.

[xmba15]

@fatihkykc I could export the (new) outdoor loftr model too, but the performance was bad. Some other issues in loftr repo also reported the same thing with the (new) outdoor loftr model. If you can test and still get a good result, it would be great. I am curious to see the result.

[fatihkykc]

@xmba15 Hi, can you provide some sample results, or at least a link to some sample images that outdoor loftr version doesnt work well, it may be a good start for comparing the results