Descriptor training result in unreasonable matches and validation loss dramatically unstable

[Merical]

I'm trying to train superpoint net by keras (tensorflow backend) and COCO2014. During the training of descriptor I found the validation loss quite unstable, and the match drawed by opencv seems a mess. Is here anyone got a reasonable output of Desc net? Thanks. I can really use some advices. Here is the outcome of my work:

[rpautrat]

Hi @Merical

I had similar problems of unstable training at the beginning when training the descriptor part of SuperPoint. However by tuning the parameters I was finally able to get a smoother training and decent matches (with similar results as with the Pretrained model of MagicLeap).

You can find the parameters I used in https://github.com/rpautrat/SuperPoint/blob/superpoint_v1/superpoint/configs/superpoint_coco.yaml. In particular, reducing the learning rate to 0.0001, setting the lambda balancing the detector and descriptor loss to 1 and the lambda_d of the original paper (balancing the number of positive / negative matches) to 800 proved to be optimal in my case. But you will probably have to tune them again for your own implementation. Good luck with that!!

[Merical]

Hi @rpautrat I've tried the parameters you suggested, and the valid loss became smoother. However, the matches are still not working. I'll try to tune the parameters. Thanks for your advice.

[rpautrat]

Good luck with the parameter tuning!

And be sure as well that there is no small bug hidden in the computation of the descriptor loss (that's what happened to me and I had like you a training rather smooth at the end, but the matches didn't work). You can for example check in a notebook on a simple example that your computations of the loss are doing exactly what you want them to do.

[Merical]

@rpautrat Thanks! I'll check the descriptor computation and match process.

[Merical]

@rpautrat Hi, I'm double-checking my matcher code. May I ask if your matcher uses the cv2.BFMatcher with NORM_L2? I tried to feed my descriptors to BFMatcher and it reported Assertion Failed.

[rpautrat]

Yes, I use the cv2.BFMatcher with NORM_L2 (and crossCheck=True). What kind of assertion failed? Are you sure that the descriptors you feed as input to the matcher have the right format?

[Merical]

@rpautrat It was a dtype bug of descriptors and I missed it. I found the matcher I used to apply output quite different matches from BFMather with NORM_L2. Maybe there are some bugs. I'm using BFMatcher and tuning my parameters. Hope for a better performance. Thanks man, it's really nice of you.

[rpautrat]

No problem :) Let me know if you manage to get a better performance!

[Merical]

@rpautrat Hi, I modified my matcher and retrained my model with the parameters you suggested. The matches became more decent! But they still contain quite a lot mismatches. This picture shows the top 32 matches in distances. I'll keep tuning my parameters and see what will happen. By the way, I found the pred extraction and desc extraction process cost quite much time. Is it possible to accelerate them?

[rpautrat]

Indeed, it's improving! There are many matches that are slightly wrong because the keypoints are not exactly at the same location, which makes me think that it is more a problem with the repeatability than with the descriptors. Maybe you should focus more on improving the repeatability.

Extracting the descriptors is indeed quite long, mainly because it has to write huge files on disk. If you want to accelerate this writing process, you can consider using an alternative to npz to store numpy arrays quicker. I think that Hdf5 is faster, or otherwise there are some fast serializer tools, like https://github.com/Blosc/Bloscpack.

[Merical]

@rpautrat Yeah I think you are right about the repeatability! I'll try to improve it, and thanks about the tips on accelerate the extracting process. You are awesome.

[UCRajkumar]

@Merical Is your keras implementation publicly available?

[Merical]

@UCRajkumar I'm organizing the code and evaluating the detector. Coming soon!!!