Pose Refiner Diverges

[FabianSchuetze]

Thank you for the wonderful work. Both the paper and the code are a pleasure to read.

I have tried the approach on a different dataset and would like to ask for your expert opinion, if I may. A fraction of the predictions (~60%) are very good even with only RGB refinement, but the remaining pose predictions are far away from the actual pose (about 1m in l1 distance) and could be ruled out by calculating the xyz boundaries of the crop. The input pose from the PNP between both fraction almost equally good.

• Do you have an idea how to discipline the refinement?
• How would you analyze the quality of the incoming query image or the sampled keys?
• Do you have any other suggestions what to look for in these cases?

Thanks again for the wonderful work.

[rasmushaugaard]

Hi, thanks for your interest in this work.

It could be caused by poor query images due to the reality gap. Have you tried to run inference on your synthetic images to see if that works? If it does, and you're able to obtain better CAD models, that might be a way to go.

[FabianSchuetze]

Thanks for your kind and informative reply.

The same happens in fact even with the training images (using the train images as test images). I tried the network not to full completion but we are already in the "almost flat" part of the learning curve. Do you have a good idea for how to evaluate the quality of query images?

[rasmushaugaard]

The nce_loss is a metric for query image / key quality.

Just out of interest, are you able to provide a representative example of where it fails? Input image, query image, key image, initial pose and pose after refinement? (The images from the infer_debug script)

[lksy0217]

I tried installing the surfemb and succeeded in proceeding with the learning

However, it failed to perform infer.py. When i run infer.py, i saw the result screen below.

I tried debugging to find the cause of the problem, and as a result, I found that code could not be performed.

I think the cause of the problem is data(infer.py)

Because in the third image, the phrase "running pose est." is printed, but the infer function is not performed, so all_pos.npy finally stores garbage values, and all_score.npy, all_timing.npy contains only initialized values

I want to help you please. Tell me what the problem is