implementation details of the network

[jcn16]

Can you describe the implementation details of the network ? how to combine the content embedding and illumination embedding in the decoder? size of the content embedding is (B,64,64,512), size of the illumination embedding is (B,9), B denotes batch size.

[heathentw]

Hi @jcn16 , do you want to discuss the implementation together?

[yangliuav]

Hi @jcn16 @heathentw Sorry for replying so late. The content embedding and illumination embedding both calculate from one embedding via a 2D Conv layer and full connection layer. For combining these two embeddings, we sum the output of a 2D layer for content embedding and a full connection layer for illumination embedding.

[heathentw]

@yangliuav Hi, thank you very much for your replay! I also got a question about the "Spherical Harmonic loss";

1. For using rotation as augmentation, the weight is [1, −1, −1, −1, 1, −1, −1, −1, 1]T, why is the fifth element not -1? since the 'xy' dimension is not changed, seems like we need to multiply -1 to achieve sum=0.
2. For using rotation as augmentation, how to C(A) guide the exchange of dimensions? for example, if we aim on the 1th and 3th dimension to exchange: [a, b, c] [1, 1, 1] + [c, b, a] [-1, -1, -1] = 0, but [a, b, c] [1, 1, 1] + [a, b, c] [-1, -1, -1] will still be zero loss......is there some magic works in spherical harmonic such that we don't need to worry about the dimension exchange part? Again thanks for your time!

[yangliuav]

1. Yes. It is a typo problem. It should be -1.
2. In fact, we have designed an exchanged loss to this cases before. For example, we hope the second element is equal to the fourth element after rotation. However, after testing, we found this loss contribution is limited. I think other augmentation is enough to handle this case.

[heathentw]

@yangliuav thanks for the reply and sharing the exchanged loss experience!

[kelisiya]

@yangliuav thanks for the reply and sharing the exchanged loss experience!

Have you implemented this algorithm ?