Can you provide some more context about the formulae in these lines? Thanks :)

zxilucky / LAGaBi

1 stars 0 forks source link

Can you provide some more context about the formulae in these lines? Thanks :) #1

Open Jordy-VL opened 11 months ago

Jordy-VL commented 11 months ago

https://github.com/zxilucky/LAGaBi/blob/de1dacba8c32126f9039bcc3b2055f96dc085df5/model/modeling_roberta.py#L245C1-L267C76

Jordy-VL commented 9 months ago

please? ^^

zxilucky commented 9 months ago

This is the official version of Roberta from Hugging Face, where we have set the 'position_embedding_type' to 'absolute' for roberta-base. Our only modification was made to the attention block at lines 266-267, where we incorporated our Gaussian bias. We will make the code more clear. Thanks.