Potential problem with not truncating sequences

matejklemen commented 4 years ago

When trying to train a Finnish model (using TDT treebank and bert-base-multilingual-uncased), I ran into a cryptic CUDA error:

  ...
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_bert.py", line 177, in forward
    embeddings = inputs_embeds + position_embeddings + token_type_embeddings
RuntimeError: CUDA error: device-side assert triggered

When trying to figure out what was wrong and running the script with CUDA_LAUNCH_BLOCKING=1, I found out some clues about it being an indexing problem (going out of bounds somewhere). This lead me to think it had something to do with BERT's limited max sequence length (512) and sure enough the script stopped crashing when I removed the example which had a length longer than that from the training set.

I unfortunately don't have a fix ready but I thought I'd leave a note here in case somebody else runs into this problem.

yzhangcs commented 4 years ago

Yes, I have some discussions in #27.

matejklemen commented 4 years ago

Oh, I see.. my bad :sweat_smile: feel free to close this then

yzhangcs / parser

Potential problem with not truncating sequences #42