Addition of 1. in the gating blocks

google-research / maxim

[CVPR 2022 Oral] Official repository for "MAXIM: Multi-Axis MLP for Image Processing". SOTA for denoising, deblurring, deraining, dehazing, and enhancement.

https://arxiv.org/abs/2201.02973

Apache License 2.0

1.02k stars 110 forks source link

Addition of 1. in the gating blocks #26

Closed sayakpaul closed 2 years ago

sayakpaul commented 2 years ago

Why is there an additional of 1 to v in the gating blocks?

An example:

https://github.com/google-research/maxim/blob/ae458a6f2be191b3b6a619feac5739ccc3ae6975/maxim/models/maxim.py#L149

@vztu

vztu commented 2 years ago

It's like adding a skip connection: x = u * v + u. Actually doesn't matter that much in terms of performance.

sayakpaul commented 2 years ago

Ah. Didn't even notice that. Pardon my stupidity.