After following the preprocessing steps, I tried running python src/generate.py -train_from [path/to/gpus_0valid_best.pt]
And got the following error:
/pytorch/torch/lib/THC/THCTensorIndex.cu:321: void indexSelectLargeIndex(TensorInfo<T, IndexType>, TensorInfo<T, IndexType>, TensorInfo<long, IndexType>, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2]: block: [38,0,0], thread: [31,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
THCudaCheck FAIL file=/pytorch/torch/lib/THC/THCTensorCopy.cu line=204 error=59 : device-side assert triggered
Traceback (most recent call last):
File "src/generate.py", line 160, in <module>
main(opt)
File "src/generate.py", line 128, in main
concept_scores,rel_scores,output =generate_graph(AmrModel,AmrDecoder,dev_data,dicts,file)
File "src/generate.py", line 47, in generate_graph
probBatch = model(srcBatch )
File "/home/USER/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 224, in __call__
result = self.forward(*input, **kwargs)
File "/home/USER/AMR_AS_GRAPH_PREDICTION/parser/models/__init__.py", line 311, in forward
probBatch,src_enc= self.concept_decoder(srcBatch)
File "/home/USER/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 224, in __call__
result = self.forward(*input, **kwargs)
File "/home/USER/AMR_AS_GRAPH_PREDICTION/parser/models/ConceptModel.py", line 135, in forward
src_enc = self.encoder(srcBatch)
File "/home/USER/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 224, in __call__
result = self.forward(*input, **kwargs)
File "/home/USER/AMR_AS_GRAPH_PREDICTION/parser/models/ConceptModel.py", line 59, in forward
ner_emb = self.ner_lut(input[:,TXT_NER])
File "/home/USER/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 224, in __call__
result = self.forward(*input, **kwargs)
File "/home/USER/.local/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 94, in forward
self.scale_grad_by_freq, self.sparse
File "/home/USER/.local/lib/python3.6/site-packages/torch/nn/_functions/thnn/sparse.py", line 41, in forward
ctx._indices = indices.contiguous()
RuntimeError: cuda runtime error (59) : device-side assert triggered at /pytorch/torch/lib/THC/THCTensorCopy.cu:204
This throws on the line ner_emb = self.ner_lut(input[:,TXT_NER])here
Upon inspection, it seems that the lookup in the pos embeddings one line above that is the culprit. There is a pos index of 46 in the data, whereas the embs["pos_lut"] is of dim 46 x 32. I assume that dim size 46 means that it allows any int in the set {0, ..., 45}
After following the preprocessing steps, I tried running
python src/generate.py -train_from [path/to/gpus_0valid_best.pt]And got the following error:
This throws on the line
ner_emb = self.ner_lut(input[:,TXT_NER])hereUpon inspection, it seems that the lookup in the pos embeddings one line above that is the culprit. There is a pos index of 46 in the data, whereas the
embs["pos_lut"]is of dim 46 x 32. I assume that dim size 46 means that it allows any int in the set {0, ..., 45}