search

karpathy / pytorch-normalizing-flows

Normalizing flows in PyTorch. Current intended use is education not production.
845 stars 98 forks source link

MADE bias note #8

Open TrentBrick opened 4 years ago

TrentBrick commented 4 years ago

Not a bug as much as an important note for MADE which are producing deltas on the input:

For the output node where all of its input connections masked, the bias for this neuron should also be masked and currently is not. The code inside MaskedLinear should be changed as follows: 


def set_mask(self, mask): # called when the masks are created. passes in this mask. 
    mask = torch.from_numpy(mask.astype(np.uint8).T)
    self.mask.data.copy_(mask)
    # if all of the inputs are zero, need to ensure the bias 
    # is zeroed out!
    self.bias_all_zero_mask = (mask.sum(dim=1)!=0).float()

def forward(self, input):
    return F.linear(input, self.mask * self.weight, self.bias_all_zero_mask * self.bias)```