Closed pkulwj1994 closed 3 years ago
Thanks, but I think this should correspond when you put a linear layer in front (such as nn.Linear
or nn.Conv2d
).
This is similar to the Gated Linear Unit which is implemented in PyTorch.
Got it! This implementation is quite similar to a non-parametric Affine Coupling Layer without output concatenation. So this kind of GLU will result in a reduction of dimension. Hope I understand correctly.
This experiments is so good and complete for generative modeling. When I walk through all codes in details I find the GatedTanhUnit implementation (summarized in GatedTanhUnit, detialed in gated_tanh) does not match my understanding for GatedTanhUnit.
The GateTanhUnit paper seems filter same input by two different streams and then elementwise multiply two filtered outputs. However in this repo, GatedTanhUnit seems divide (chunk) the input by certain dimension, and then filter two parts and elementwise multiply the results. The final output will be half the dimensional of input. Detailed codes are copied below
Great Thanks to check and response!