search

FangjinhuaWang / PatchmatchNet

Official code of PatchmatchNet (CVPR 2021 Oral)
MIT License
503 stars 70 forks source link

A few questions about the code #28

Closed fuy34 closed 3 years ago

fuy34 commented 3 years ago

Hi,

Thank you for releasing the code. I really like this work, and I am trying to understand the code. Here are a few questions.

  1. Why the depth initialization and disruption is set in the disparity space? Is there any drawback you found to directly operate in the depth space?
  2. Could you offer more insights of feature_weight, depth_weight , and view_weight in patchmatch.py?
    • feature_weight and view_weight seem to be learnable weights given the similarity between center pixel feature and neighbor's feature and ref_feature and src_feature, respectively. Shall we expect the higher weight of similar pixels or opposite?
    • depth_weight offers higher weights for pixels with similar depth to the center pixel. But why is this needed for the evaluation process?
    • How effective are these weights? Do traditional patch match methods have any of these weights?
  3. Is the depth_internal_scale related to the R_k in the paper? If yes, may I ask what the relationship is?
  4. For score in patchmatch.py (line 197), why shall we take its exponential after softmax?

Please excuse me for the long list. Looking forward to learning more about your work.

FangjinhuaWang commented 3 years ago

Hi,

  1. this is more suitable for large scale scene (explained in 3.2.1, there are also some references there).

  2. (1) view weight is used to set different weights for different source images since some pixels may be occluded in some images but not occluded in the others, the idea is widely used in traditional methods (COLMAP, ACMM) and also used in learning-based methods (PVSNet, PVA-MVSNet) (2) feature weight descirbes the similarity, so similar pixels should have higher weight (3) as explained in supplementary sec 1, along the depth dimension, different pixels have different depth hypotheses. So we use a depth weight to balanced their contribution (if the difference is too large, then the contribution is small)

  3. R_k = N_k * (depth_internal_scale -1)
  4. i first use logsoftmax and then take the exp for numerical stability. actually, i think using original softmax is enough
fuy34 commented 3 years ago

Thank you. These are very helpful!