How to modify stbn while rotating patch

[weilonghu]

I want to add rotated patches during the patch matching process, and found that the prepareExamplePatches function in the patch_matching_and_quilting.py file has the following code:

def prepareExamplePatches(self):
    print('Preparing example patches ...')
    result = self.patches
    stbn = self.sample_tbn

    print(f"patches shape: {result.shape}")
    print(f"stbn shape: {stbn.shape}")

    self.total_patches_count = result.shape[0]
    if self.mirror_hor:
        hor_result = result[:, ::-1, :, :]
        result = np.concatenate((result, hor_result))
        hor_stbn = np.copy(stbn)
        hor_stbn[..., 0, :] *= -1
        stbn = np.concatenate([stbn, hor_stbn], axis=0)
    if self.mirror_vert:
        vert_result = result[:, :, ::-1, :]
        result = np.concatenate((result, vert_result))
        hor_vtbn = np.copy(stbn)
        hor_vtbn[..., 1, :] *= -1
        stbn = np.concatenate([stbn, hor_vtbn], axis=0)
    if self.rotate:
        rot_result1 = np.rot90(result, 2)
        rot_result2 = np.rot90(rot_result1, 2)
        rot_result3 = np.rot90(rot_result2, 2)
        result = np.concatenate((result, rot_result1, rot_result2, rot_result3))
    return result, stbn

I have the following questions:

1. What is the function of stbn？
2. The shape of stbn is (n, 9), is only one line ofstbn transformed in the mirror_hor and mirror_vert code branches?
3. In the rotate code branch, how to change stbn？

Thanks

[yihua7]

Thanks for your questions and I found a bug here.

1. stbn is the tangents, bi-tangents, and normals of sampled frames. We record it to adjust the lighting model when rendering synthesized textures. Note that we need not only to record the stbn of sampled frames but also to apply horizontal and vertical transformations to them when performing corresponding patch augmentations. For more details, please refer to the $T_s$ mentioned in Sec.3.2.1 and Eq.4 in our paper.
2. stbn should be of shape (n, 3, 3). The x and y axis are flipped when doing horizontal and vertical mirror augmentation respectively. Current operations on the shape (n, 9) is a bug and should be fixed in the future.
3. In the rotate branch, stbn should also be rotated correspondingly. But I never rotated them and should had deleted the branch in the released codes.

[yihua7]

Bug fixed in the commit.