search

hila-chefer / Transformer-Explainability

[CVPR 2021] Official PyTorch implementation for Transformer Interpretability Beyond Attention Visualization, a novel method to visualize classifications by Transformer based networks.
MIT License
1.75k stars 232 forks source link

Does def safe_divide really work? #67

Open 686llhhhhhh opened 6 months ago

686llhhhhhh commented 6 months ago 
def safe_divide(a, b):
    den = b.clamp(min=1e-9) + b.clamp(max=1e-9)  # ???
    den = den + den.eq(0).type(den.type()) * 1e-9
    return a / den * b.ne(0).type(b.type())

and 

b = torch.randn((1, 10))
print((b.clamp(min=1e-9) + b.clamp(max=1e-9)) == (b + 1e-9))
# output = tensor([[True, True, True, True, True, True, True, True, True, True]])