When trying to train a Finnish model (using TDT treebank and bert-base-multilingual-uncased), I ran into a cryptic CUDA error:
...
File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_bert.py", line 177, in forward
embeddings = inputs_embeds + position_embeddings + token_type_embeddings
RuntimeError: CUDA error: device-side assert triggered
When trying to figure out what was wrong and running the script with CUDA_LAUNCH_BLOCKING=1, I found out some clues about it being an indexing problem (going out of bounds somewhere). This lead me to think it had something to do with BERT's limited max sequence length (512) and sure enough the script stopped crashing when I removed the example which had a length longer than that from the training set.
I unfortunately don't have a fix ready but I thought I'd leave a note here in case somebody else runs into this problem.
When trying to train a Finnish model (using TDT treebank and bert-base-multilingual-uncased), I ran into a cryptic CUDA error:
When trying to figure out what was wrong and running the script with
CUDA_LAUNCH_BLOCKING=1
, I found out some clues about it being an indexing problem (going out of bounds somewhere). This lead me to think it had something to do with BERT's limited max sequence length (512) and sure enough the script stopped crashing when I removed the example which had a length longer than that from the training set.I unfortunately don't have a fix ready but I thought I'd leave a note here in case somebody else runs into this problem.