-
in TinyBERT/task_distill.py line 973:
``` python
elif output_mode == "regression":
loss_mse = MSELoss()
cls_loss = loss_mse(student_logits.view(-1), label_ids.view(-1))
```
so TinyBERT i…
-
Thank you for sharing this repo.
I have a question about the loss used to train TinyBert. Unlike DistilBert, MobileBert and other distillation based BERT variants, TinyBert training doesn't include t…
-
I got the following error with Select API for RAG ollama:
DEBUG:matplotlib.pyplot:Loaded backend tkagg version 8.6.
DEBUG:matplotlib.pyplot:Loaded backend agg version v2.2.
ERROR:root:Using embed…
-
Hello! The work is great and thanks for sharing the codes!
But I am confused about the general distill stage: In the Bert-emd folder, I see a file called "general_distill" but it seems that the fil…
-
设置了reduce_memory,运行到filename=self.working_dir/'input_ids.memmap' 这里报错“另一个程序正在使用此文件,进程无法访问”
先改成 filename=str(self.working_dir/'input_ids.memmap'), 第一个epoch没问题,第二次input_ids = np.memmap报错
再按照pregen…
-
python: 3.7
transformers: 4.9.2
pytorch: 1.8.1
```python
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("huawei-noah/TinyBERT_4L_zh")
model = AutoM…
-
你好请问一下tinybert的训练方式是不是直接从最原始的开始训练呢?还是从大模型distill到小模型呢?
-
if i want import my chinese data in TinyBert ,what should I do?
-
看了tinybert的代码只有torch的,有tf版本的吗
-
hello,我是用tinybert做韵律预测任务,我用中文的预训练任务做数据加强后,得到的数据感觉有问题,是不能用于韵律预测的,请问哪数据增强是不是不用了