search

Lisennlp / TinyBert

简洁易用版TinyBert：基于Bert进行知识蒸馏的预训练语言模型
251 stars 49 forks source link

Can I use it to distill roberta? #3

Closed Tweakzx closed 3 years ago

Tweakzx commented 3 years ago

I used it to distill a roberta model, but there are some errors the may be "out of the index" thing?I am confused if tinybert can distill roberta? Hope you help me .

/opt/conda/conda-bld/pytorch_1565272279342/work/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [47,0,0], thread: [63,0,0] Assertion srcIndex < srcSelectDimSize failed.

Lisennlp commented 3 years ago

Yes, your problem should be that your data length exceeds Bert’s position limit of 512.