search

tatp22 / linformer-pytorch

My take on a practical implementation of Linformer for Pytorch.
https://arxiv.org/pdf/2006.04768.pdf
MIT License
400 stars 36 forks source link

embeddings_mask datatype #14

Closed tongcu closed 3 years ago

tongcu commented 3 years ago

Is embeddings_mask a bool variable in your data？

tatp22 commented 3 years ago

Hi @tongcu!

The embeddings_mask is supposed to be a torch.BoolTensor type. Basically, it is a torch tensor that is filled with boolean values, and the inputs that are False are the ones that are masked out.

The way it works is that it is the same length as the input, except that on the input size dimension, it is supposed to be filled with booleans instead of embedding tokens (in the case of the of the LinformerLM module) or vectors (for Linformer).