Question about homo_warping

[CaptainEven]

    rot_depth_xyz = rot_xyz.unsqueeze(2).repeat(1, 1, num_depth, 1) * depth_values.view(batch, 1, num_depth,
                                                                                        1)  # [B, 3, N_depth, H*W]
    proj_xyz = rot_depth_xyz + trans.view(batch, 3, 1, 1)  # [B, 3, N_depth, H*W]

Is the tanslation missing multiplication with depth values here according to the plane-induced homography?

[kwea123]

No, translation doesn't need depth values. Originally it is a matrix multiplication of 4x4 (projection matrix) times 4x1 (3d coordinate, homogeneous) as follows:

[R T] [xd]
[0 1] [yd]
      [d]
      [1]

= [R][xd] + [T]
     [yd]
     [d]

So if you do the multiplication, the parts with d inside is only multiplied with R.

[JiangQi715]

@kwea123 In the Homo warming function, the normalization result after coordinate transformation may not be between - 1 and 1. How do you understand this? I look forward to your reply, thank you.

[kwea123]

If it's not between -1 and 1 it means that pixel is warped outside of the image, in that case that pixel has value 0 (all black) which is specified by padding_mode='zeros'

[whubaichuan]

@JiangQi715 See the usage of grid_sample

[JiangQi715]

@kwea123 OK, I see what you mean. Now I have a problem: all the pixels are warped out of the image. What may be the problem? Is the previous matrix operation wrong (resulting in a warp error)?

[JiangQi715]

@whubaichuan 您好，这个grid_sample我用起来了。是在这个grid_sample之前遇到了一个问题：homo_warming函数中，像素坐标变换（旋转平移）后，x和y坐标变大了（超过了宽度W和高度H），使用这个代码的归一化方法后，导致grid全部在[-1,1]之外。请问可能是哪里出了问题？是不是前面的矩阵运算出错了（导致坐标变换错误）？还是这个归一化方法不适用呢？十分谢谢。

[JiangQi715]

@whubaichuan 您好，这个grid_sample我用起来了。是在这个grid_sample之前遇到了一个问题：homo_warming函数中，像素坐标变换（旋转平移）后，x和y坐标变大了（超过了宽度W和高度H），使用这个代码的归一化方法后，导致grid全部在[-1,1]之外。请问可能是哪里出了问题？是不是前面的矩阵运算出错了（导致坐标变换错误）？还是这个归一化方法不适用呢？十分谢谢。

代码在MVSNet_pytorch/models/module.py的117-123行

[kwea123]

你用的是DTU dataset?這是不可能的，這個代碼我確認過了是正確的，不會全部在[-1, 1]之外的。 還是你用的是別的dataset?

[JiangQi715]

那可能是我的DTU数据集预处理部分出了一些问题，我自己再检查一下。再次感谢您。

[whubaichuan]

@JiangQi715 是不是用到了torch.meshgrid？。注意其和np.meshgrid和tf.meshgrid的区别

[JiangQi715]

@JiangQi715 是不是用到了torch.meshgrid？。注意其和np.meshgrid和tf.meshgrid的区别

是的，用到了，我查一下。谢谢您。

[czha5168]

Hi CoolCats[0] Nice to see your discussions above, as I am new to this domain and your discussions really help! Regarding the homo_warping function(), I do have one more concern and hoping to learn from your ideas on this. Any comments are appreciated! In https://github.com/xy-guo/MVSNet_pytorch/blob/e0f2ae3d7cb2dd13807b775f2075682eaa7f1521/models/module.py#L106, why we set torch.no_grad()? Is it against the differential design behind the idea of the differential homo warping? Thanks in advance and looking forward to your replies! Take care.

[whubaichuan]

@czha5168 In fact, the code "with torch.no_grad()" is redundant. Cause that the grid will not pass the grad back. So if you delete the code "with torch.no_grad()", it's no different.

[czha5168]

@czha5168 In fact, the code "with torch.no_grad()" is redundant. Cause that the grid will not pass the grad back. So if you delete the code "with torch.no_grad()", it's no different.

Hi Baichuan! Thanks so much for your reply!

[UestcJay]

@JiangQi715 是不是用到了torch.meshgrid？。注意其和np.meshgrid和tf.meshgrid的区别

Hi,recently I have a problemin homo_warp, If there is occlusion in the sampling place, the wrong place will be sampled, have you considered this problem? can you tell me how to solve？