Documentation for Masks

[calebh]

I see in some of the issues in this repo that masking is supported in this package, but I don't see it documented anywhere. What type of masking is supported, how do I use it, and does it match the PyTorch API? See https://pytorch.org/docs/master/nn.html#torch.nn.Transformer.forward

Do you have a function to automatically generate the square subsequent mask?

Thanks for making this package.

[lucidrains]

@calebh Hi Caleb! Thanks for your interest in this repository. Here you go! https://github.com/lucidrains/reformer-pytorch/commit/4d7217e882bcbec5283f5bcd8947940609711269

[calebh]

Is the context mask a reformer specific mask? I don't see an equivalent mask parameter in the official PyTorch transformer documentation. Also you say that input_attn_mask is rarely used, but surely this parameter would be used all the time in the decoder?

Also regarding what True/False means in the mask. It seems in the official PyTorch implementation of the transformer, positions with a value of True will be ignored, and False will be unchanged (see the link in my first comment). Just to clarify, this is the opposite of what your package uses?

Also does the ReformerEncDec class take the same parameters for masking? Do I have to generate and pass in the triangular attention mask to the decoder portion myself?

[lucidrains]

@calebh I did not use the official Pytorch implementation as reference

You don't need to worry about the causal triangular mask, you just have to set causal = True for it to be added.

ReformerEncDec has the exact same parameters as the ReformerLM class except you prepend the encoder parameters as enc_ and the decoder as dec_

[calebh]

Great! Thanks for the help 👍

[lucidrains]

@calebh no problem! hope you find this library useful!