search

dropreg / R-Drop

870 stars 107 forks source link

Where is R-Drop code in R-Drop/huggingface_transformer_src/bert_rdrop/run_glue.py? #18

Closed zhenshiqi1996 closed 3 years ago

TxiHu commented 2 years ago

Can you find the code for the two dropouts?

ruoyusong commented 2 years ago

i also have this question

TxiHu commented 2 years ago

i also have this question

我觉得两次dropout其实就是把同一个输入输入到模型两次。两次会产生一些微小差别。

ruoyusong commented 2 years ago

i also have this question

我觉得两次dropout其实就是把同一个输入输入到模型两次。两次会产生一些微小差别。

那KL散度呢 源码里没找打哎

TxiHu commented 2 years ago

i also have this question

我觉得两次dropout其实就是把同一个输入输入到模型两次。两次会产生一些微小差别。

那KL散度呢 源码里没找打哎

它文章里的Usage里已经写了损失计算公式呀。

XiaoqingNLP commented 2 years ago

写在了modeling_bert.py 里面bert_kl 函数就是