Hi,
I am always getting the error while trying to run the training with Classifier Encoder. Following is the error log.
[2019-12-13 14:23:06,335 INFO] number of parameters: 109483009
[2019-12-13 14:23:06,336 INFO] Start training...
[2019-12-13 14:23:06,674 INFO] Loading train dataset from ../bert_data/cnndm.train.123.bert.pt, number of examples: 2001
Better speed can be achieved with apex installed from https://www.github.com/nvidia/apex.
gpu_rank 0
Traceback (most recent call last):
File "train.py", line 340, in
train(args, device_id)
File "train.py", line 272, in train
trainer.train(train_iter_fct, args.train_steps)
File "/mnt/nfs/scratch1/riteshkumar/BertSum/src/models/trainer.py", line 157, in train
report_stats)
File "/mnt/nfs/scratch1/riteshkumar/BertSum/src/models/trainer.py", line 321, in _gradient_accumulation
sent_scores, mask = self.model(src, segs, clss, mask, mask_cls)
File "/mnt/nfs/work1/mfiterau/riteshkumar/miniconda3/envs/nlp/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(input, kwargs)
File "/mnt/nfs/scratch1/riteshkumar/BertSum/src/models/model_builder.py", line 93, in forward
top_vec = self.bert(x, segs, mask)
File "/mnt/nfs/work1/mfiterau/riteshkumar/miniconda3/envs/nlp/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, *kwargs)
File "/mnt/nfs/scratch1/riteshkumar/BertSum/src/models/model_builder.py", line 52, in forward
encodedlayers, = self.model(x, segs, attention_mask =mask)
File "/mnt/nfs/work1/mfiterau/riteshkumar/miniconda3/envs/nlp/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(input, kwargs)
File "/mnt/nfs/work1/mfiterau/riteshkumar/miniconda3/envs/nlp/lib/python3.7/site-packages/pytorch_pretrained_bert/modeling.py", line 711, in forward
embedding_output = self.embeddings(input_ids, token_type_ids)
File "/mnt/nfs/work1/mfiterau/riteshkumar/miniconda3/envs/nlp/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, *kwargs)
File "/mnt/nfs/work1/mfiterau/riteshkumar/miniconda3/envs/nlp/lib/python3.7/site-packages/pytorch_pretrained_bert/modeling.py", line 258, in forward
position_embeddings = self.position_embeddings(position_ids)
File "/mnt/nfs/work1/mfiterau/riteshkumar/miniconda3/envs/nlp/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(input, **kwargs)
File "/mnt/nfs/work1/mfiterau/riteshkumar/miniconda3/envs/nlp/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 117, in forward
self.norm_type, self.scale_grad_by_freq, self.sparse)
File "/mnt/nfs/work1/mfiterau/riteshkumar/miniconda3/envs/nlp/lib/python3.7/site-packages/torch/nn/functional.py", line 1506, in embedding
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: index out of range at /opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/TH/generic/THTensorEvenMoreMath.cpp:193
Hi, I am always getting the error while trying to run the training with Classifier Encoder. Following is the error log. [2019-12-13 14:23:06,335 INFO] number of parameters: 109483009 [2019-12-13 14:23:06,336 INFO] Start training... [2019-12-13 14:23:06,674 INFO] Loading train dataset from ../bert_data/cnndm.train.123.bert.pt, number of examples: 2001 Better speed can be achieved with apex installed from https://www.github.com/nvidia/apex. gpu_rank 0 Traceback (most recent call last): File "train.py", line 340, in
train(args, device_id)
File "train.py", line 272, in train
trainer.train(train_iter_fct, args.train_steps)
File "/mnt/nfs/scratch1/riteshkumar/BertSum/src/models/trainer.py", line 157, in train
report_stats)
File "/mnt/nfs/scratch1/riteshkumar/BertSum/src/models/trainer.py", line 321, in _gradient_accumulation
sent_scores, mask = self.model(src, segs, clss, mask, mask_cls)
File "/mnt/nfs/work1/mfiterau/riteshkumar/miniconda3/envs/nlp/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward( input, kwargs)
File "/mnt/nfs/scratch1/riteshkumar/BertSum/src/models/model_builder.py", line 93, in forward
top_vec = self.bert(x, segs, mask)
File "/mnt/nfs/work1/mfiterau/riteshkumar/miniconda3/envs/nlp/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, *kwargs)
File "/mnt/nfs/scratch1/riteshkumar/BertSum/src/models/model_builder.py", line 52, in forward
encodedlayers, = self.model(x, segs, attention_mask =mask)
File "/mnt/nfs/work1/mfiterau/riteshkumar/miniconda3/envs/nlp/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(input, kwargs)
File "/mnt/nfs/work1/mfiterau/riteshkumar/miniconda3/envs/nlp/lib/python3.7/site-packages/pytorch_pretrained_bert/modeling.py", line 711, in forward
embedding_output = self.embeddings(input_ids, token_type_ids)
File "/mnt/nfs/work1/mfiterau/riteshkumar/miniconda3/envs/nlp/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, *kwargs)
File "/mnt/nfs/work1/mfiterau/riteshkumar/miniconda3/envs/nlp/lib/python3.7/site-packages/pytorch_pretrained_bert/modeling.py", line 258, in forward
position_embeddings = self.position_embeddings(position_ids)
File "/mnt/nfs/work1/mfiterau/riteshkumar/miniconda3/envs/nlp/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(input, **kwargs)
File "/mnt/nfs/work1/mfiterau/riteshkumar/miniconda3/envs/nlp/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 117, in forward
self.norm_type, self.scale_grad_by_freq, self.sparse)
File "/mnt/nfs/work1/mfiterau/riteshkumar/miniconda3/envs/nlp/lib/python3.7/site-packages/torch/nn/functional.py", line 1506, in embedding
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: index out of range at /opt/conda/conda-bld/pytorch_1556653114079/work/aten/src/TH/generic/THTensorEvenMoreMath.cpp:193