search

localminimum / QANet

A Tensorflow implementation of QANet for machine reading comprehension
MIT License
983 stars 310 forks source link

InvalidArgumentError during evaluation #27

Closed maciejbiesek closed 5 years ago

maciejbiesek commented 6 years ago

Hello,

for some questions in SQuAD dataset I got exception:

InvalidArgumentError (see above for traceback): num_upper must be negative or less or equal to number of columns (10) got: 30 [[Node: Output_Layer/MatrixBandPart = MatrixBandPart[T=DT_FLOAT, Tindex=DT_INT64, _device="/job:localhost/replica:0/task:0/device:CPU:0"](Output_Layer/MatMul, Output_Layer/MatrixBandPart/num_lower, Output_Layer/MatrixBandPart/num_upper)]]

Do you know what is the reason for that? How to get rid of this problem?

localminimum commented 6 years ago

Hi @maciejbiesek , it seems that the batch you are evaluating contains an example that has a context length less than 30. A simple solution is decreasing the matrix band part size, however it may result in change in performance. To properly address this problem, use a batch with all examples having the context length of at least the size of the matrix band part. Thanks

maciejbiesek commented 6 years ago

But this problem even occurs during evaluation on dev set of SQuAD, question by question. How do you deal with that? I mean exceptions during evaluation on this dataset. Because there are fragments with less than 30 tokens.

localminimum commented 6 years ago

Hi @maciejbiesek , I haven't been successful in replicating this issue. Could you please tell me what setup you used? And how you obtained the error above. Thanks

maciejbiesek commented 6 years ago

I just wrote script to iter SQuAD json paragraph by paragraph, question by question and evaluate model on pairs (question, context). Because the length of a answer is limited to 30 tokens, if passage has less than 30, exception occurs. After changing its to 15 tokens, everything works.

DarrenAbramson commented 6 years ago

@maciejbiesek did you change ans_limit to 15 in config.py after training?

My understanding is that the model cannot accommodate contexts that are shorter than ans_limit. Would appreciate any suggestions for padding short contexts after training.