search

castorini / DeeBERT

DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
Apache License 2.0
151 stars 23 forks source link

RuntimeError: Boolean value of Tensor with more than one value is ambiguous #8

Closed quanhoang288 closed 2 years ago

quanhoang288 commented 3 years ago

Hi there I was using your model for my new classification task on Japanese text so I added the configuration needed for Japanese Bert I'm doing the training on the BertForSequenceClassifcation model and when I do validation I encountered a bug that said the if condition was done on a tensor with more than one value (that tensor is actually the highway entropy of a batch) so that threw a runtime error when compared with the early exit threshold Can you help me with this ? I'd appreciate your help on this problem Here is the snippet of the error code image

ji-xin commented 3 years ago

Hi, sorry for the late reply. I guess you are doing inference with a batch_size greater than 1? In our implementation the inference batch_size can only be 1, and the reason is exactly the issue you descirbed.

JulesBelveze commented 3 years ago

Hey @ji-xin I'm facing the same problem than @quanhoang288. Any idea how what could be a relevant way to adapt it to handle batches with size greater than 1?

ji-xin commented 3 years ago

I'm afraid there isn't a straightforward way to do it. It's certainly possible: given a layer, feed in a batch of examples, perform early exiting on each of them individually, and regroup the low-confidence examples into a new batch that will be fed to the next layer. However, this will require a bit of engineering efforts.