"Index out of bounds" error running "ac_neusum"

[ronaldahmed]

Hi Wendy,

Really great work, thank you for sharing the code. I am replicating the results using the provided pretrained models and preprocessed datasets. The system ExtrLG + NeuSum decoder is giving the following error when running test.py with option "--model ac_neusum" on the PubMed's test set :

ambd=0.6, mlp_size=100, model='ac_neusum', modelpath='<model path>', remove_stopwords=False, sent_length_limit=1000, stemmer=False, use_mmr=False, use_newloss=False, use_rl=False, use_tri
gram_block=False, word_length_limit=200)
Start loading model.
pubmed_ac_neusum
Start evaluating.
Batch 1, Loss: 0.274923
/opt/conda/conda-bld/pytorch_1614378062065/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [110,0,0], thread: [0,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378062065/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [110,0,0], thread: [1,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378062065/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [110,0,0], thread: [2,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
....
/opt/conda/conda-bld/pytorch_1614378062065/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [30,0,0], thread: [62,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1614378062065/work/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [30,0,0], thread: [63,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
Traceback (most recent call last):
  File "test.py", line 146, in <module>
    _, l, results = eval_seq2seq(test_dataloader,model,hyp_path,ref_path, word_length_limit, sent_length_limit,\
  File "/home/rcardena/kvdsum-extr/extrlg/run.py", line 293, in eval_seq2seq
    summaryfiles,referencefiles,loss,num_data,select_ids,oracle,sections = eval_seq2seq_batch_neusum(data, model,hyp_path,ref_path, \
  File "/home/rcardena/kvdsum-extr/extrlg/run.py", line 447, in eval_seq2seq_batch_neusum
    doc_sent_scores = model(document,input_length,indicators,begin,end,device,max(40,max_step),selections,doc_sent_mask)
  File "/home/rcardena/miniconda3/envs/extrlg/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/rcardena/kvdsum-extr/extrlg/models.py", line 439, in forward
    scores = self.pointer(
  File "/home/rcardena/miniconda3/envs/extrlg/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/rcardena/kvdsum-extr/extrlg/models.py", line 551, in forward
    doc_sent_mask[j][max_idx[j]] = 0
RuntimeError: CUDA error: device-side assert triggered

This error only appears for model=ac_neusum. So far I have run the other model options without problems. Could it be that there was a bug during preprocessing data for NeuSum? Would you please look into this or have an idea of what might be the cause?

I really appreciate any help you can provide. Best, Ronald