How to calculate the jacobian

[zwb0]

Hi, thanks for your novel work, and I'm confused about how to calculate the last Jacobian. For example, in the class HaarDownsampling, the Jacobian is calculated as follows,

if not rev: self.elements = x.shape[1] * x.shape[2] * x.shape[3] self.last_jac = self.elements / 4 * np.log(1/16.) ... else: self.elements = x.shape[1] * x.shape[2] * x.shape[3] self.last_jac = self.elements / 4 * np.log(16.) And in the class InvBlockExp, def jacobian(self, x, rev=False): if not rev: jac = torch.sum(self.s) else: jac = -torch.sum(self.s) return jac / x.shape[0]

I want to replace the HaarDownsampling with other neural networks, should the kernel follow some special design? And could you give me some tips on how to calculate the Jacobian of this new network? Thanks in advance! Sincerely

[pkuxmq]

In this work it is actually not needed to calculate the Jacobian. The determinate of the Jacobian matrix is calculated when we need to explicitly compute the distribution transformation (please refer to flow-based generative models).

Each module in the model should be invertible, and a usual neural network may not be invertible. You may refer to works on invertible neural networks, e.g. flow-based models like RealNVP, Glow and follow-up works, or invertible residual networks etc., to get some ideas on how to design invertible architecture.

[zwb0]

Thanks for your reply! I've read several papers related to invertible network, and they usually use the determinant of Jacobian as Loss. So in this work, we don't need to calculate the Jacobian as loss and use a discriminator instead? And I just need to design a network which can be gone through in two directions, right? Like specifying the kernel in Haar transformation as you do. Thanks a lot!

[pkuxmq]

Yes, we do not use maximum likelihood estimation as our distribution objective. Haar transformation is a bijective wavelet transformation. If you want to use a learnable neural network, it should have some design to enable invertibility.

[toddwyl]

I don't understand why your jacobian calculate by the sum of s. And the determinate of the Jacobian matrix should not be the sum of s. Hope for reply. @pkuxmq

[pkuxmq]

@ManWingloeng First, the sum of s is log det Jacobian. Second, the expression of coupling layers here is actually the combination of two layers: the first z1 = x1 + F(x2), z2 = x2 and the second y1 = z1, y2 = z2 * e^s + G(y1); the Jacobian of both layers are triangular and easy to compute, and the total Jacobian is the product of these two Jacobians. So the total log det Jacobian is the sum of two log det Jabocians, which are 0 and the sum of s respectively. More details you may refer to RealNVP and FrEIA (https://github.com/VLL-HD/FrEIA).

[toddwyl]

@pkuxmq Thanks a lot !!!! 👍 Now I think I know. BTW, do you try some loss with log det Jacobian?

[pkuxmq]

Please refer to https://github.com/pkuxmq/Invertible-Image-Rescaling/issues/20.