The masks are all equal to 1

[zxw866]

result = []
for item in items:

    if len(item) < l:

        item = [1. for _ in item] + ([0.] * (l - len(item)))

    if len(item) >= l:

        item = [1. for _ in item[:l]]
    result.append(item)

in data.py, line 148.

This code causes the masks to be all equal to 1.0 ? Is there a problem here?

[seanmacavaney]

Ah, yes it seems so. The problem arose from re-using the item variable. Thanks!

In [1]: import torch                                                                                                                                                                                        

# BEFORE

In [2]:  
   ...: def _mask(items, l): 
   ...:     result = [] 
   ...:     for item in items: 
   ...:         if len(item) < l: 
   ...:             item = [1. for _ in item] + ([0.] * (l - len(item))) 
   ...:         if len(item) >= l: 
   ...:             item = [1. for _ in item[:l]] 
   ...:         result.append(item) 
   ...:     return torch.tensor(result).float().cuda() 
   ...:                                                                                                                                                                                                     
In [3]: items = [[1,2,3,4,5,6], [1,2,3], [1], []]
In [4]: _mask(items, 3)                                                                                                                                                                                     
Out[4]: 
tensor([[1., 1., 1.],
        [1., 1., 1.],
        [1., 1., 1.],
        [1., 1., 1.]], device='cuda:0')

# AFTER

In [5]: 
    ...: def _mask(items, l): 
    ...:     result = [] 
    ...:     for item in items: 
    ...:         if len(item) < l: 
    ...:             mask = [1. for _ in item] + ([0.] * (l - len(item))) 
    ...:         if len(item) >= l: 
    ...:             mask = [1. for _ in item[:l]] 
    ...:         result.append(mask) 
    ...:     return torch.tensor(result).float().cuda() 
In [6]: _mask(items, 3)                                                                                                                                                                                    
Out[6]: 
tensor([[1., 1., 1.],
        [1., 1., 1.],
        [1., 0., 0.],
        [0., 0., 0.]], device='cuda:0')

[cmacdonald]

Sean, can you comment on which cases this will affect, and if effectiveness will be markedly affected?

[seanmacavaney]

This effects cases in which the batch is larger than 1. With the default settings, this is the case for training (gradient accumulation batch size of 2) and evaluation (batch size of 16). It will have the most effect when document lengths differ considerably (i.e., a very short document in a batch with a long document).

As far as the effectiveness goes, I will need to run some experiments to test this. Since only padding tokens are masked, I imagine that the models should learn to ignore these tokens. But the only way to know with certainty is to verify it empirically.