Descriptor interpolate

[virtualRooom]

Hello, ErikOrjehag.

You seem to be the first open source Unsuperpoint implementation on github. Nice work!

I have a question in getting the descriptor map. The paper says "The model use all point positions in Pmap to interpolate all entries in descriptor map Fmap" , Is it implemented in your forward function？If not, have you tested the descriptors and what are the results?

https://github.com/ErikOrjehag/sfmnet/blob/5b4001b3950937f604bd394ec2bb14199c1c56d7/networks/unsuperpoint.py#L118-L148

[ErikOrjehag]

Hi @virtualRooom thanks! I must have missed that part in the paper, because I don't interpolate the descriptor map. Maybe this can be achived using grid_sample like I do for depth reconstruction here:

https://github.com/ErikOrjehag/sfmnet/blob/5b4001b3950937f604bd394ec2bb14199c1c56d7/reconstruction.py#L69

I have to try that! The results I have been getting so far is promising but not fantastic. This is ongoing work for my masters thesis.

I dont have any metrics for unsuperpoint setup yet so I only evaluate by eye and dont have any hard numbers. I have not seen any other opensource unsuperpoint implementation yet so Im still in experimental phase where Im not sure if the programming is correct.... The decorrelation loss was especially tricky to implement based on the equation in the paper, maybe you have any thoughts about if I got it right? Its nice to have some feedback on the code because its very easy to make misstakes such as forgetting to sample the decriptor map like i did.

https://github.com/ErikOrjehag/sfmnet/blob/5b4001b3950937f604bd394ec2bb14199c1c56d7/networks/unsuperpoint.py#L40-L52

[virtualRooom]

In "unsuperpoint" paper, I think there are typos in the calculation of the entries of the correlation matrix.

Your decorrelate() function should work as intended, but I don't know if the last step of assignment(e.g. R[:,idx,idx] = 0 ) will cause gradient problem.

Looking forward to your future work ; )