Closed sbwww closed 2 years ago
You are right that bmodel_output
is not actually used and all we need is pooler_output
. I wrote it in this way mainly because I wanted them to align with the output of BertPooler
, BertModel
, etc., at the time of implementation.
Greetings, DeeBERT is really a crucial and easy-to-understand achievement in BERT inference acceleration.
However, in
transformers/modeling_highway_bert.py
, theforward
function of classBertHighway
is a bit confusing. Your original code is as followsI am not quite sure about introducing
bmodel_output
for it's not actually used.It seems that
pooler_input
issequence_output
, andpooled_output
is equivalent topooler_output
.Is there any trick that should be noticed? Maybe the comments starting with "return" can be updated for more details.