Closed sameerkhurana10 closed 5 years ago
Hi, I haven't seen this before. It's strange that it happens when you exclude UNKs from the perplexity computation. Could it mean that there are only UNKs in some sequence, resulting in sequence length 0 and an infinite value? To debug this further, you would need to enable debug logging (--log-level debug
) to see where the exception occurs.
It is possible that the test sentence consists on only UNKs.
okay. Does the below tell us anything?
Using cuDNN version 6021 on context None
Mapped name None to device cuda: GeForce GTX TITAN Black (0000:03:00.0)
2018-04-08 17:46:49,062 exception_handler: An unexpected TypeError exception occurred: ufunc 'isinf' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''
Traceback will be written to debug log (enable with --log-level debug).
2018-04-08 17:46:49,062 exception_handler: Traceback:
2018-04-08 17:46:49,089 exception_handler: File "/data/sls/u/sameerk/anaconda3/envs/theano-lm/bin/theanolm", line 147, in <module>
main()
2018-04-08 17:46:49,089 exception_handler: File "/data/sls/u/sameerk/anaconda3/envs/theano-lm/bin/theanolm", line 88, in main
args.command_function(args)
2018-04-08 17:46:49,089 exception_handler: File "/data/sls/u/sameerk/anaconda3/envs/theano-lm/lib/python3.5/site-packages/theanolm/commands/score.py", line 122, in score
args.output_file, args.log_base, args.subwords, False)
2018-04-08 17:46:49,089 exception_handler: File "/data/sls/u/sameerk/anaconda3/envs/theano-lm/lib/python3.5/site-packages/theanolm/commands/score.py", line 209, in _score_text
num_zeroprobs += sum(numpy.isneginf(lp) for lp in merged_logprobs)
2018-04-08 17:46:49,089 exception_handler: File "/data/sls/u/sameerk/anaconda3/envs/theano-lm/lib/python3.5/site-packages/theanolm/commands/score.py", line 209, in <genexpr>
num_zeroprobs += sum(numpy.isneginf(lp) for lp in merged_logprobs)
2018-04-08 17:46:49,089 exception_handler: File "/data/sls/u/sameerk/anaconda3/envs/theano-lm/lib/python3.5/site-packages/numpy/lib/ufunclike.py", line 34, in func
return f(x, out=out, **kwargs)
2018-04-08 17:46:49,089 exception_handler: File "/data/sls/u/sameerk/anaconda3/envs/theano-lm/lib/python3.5/site-packages/numpy/lib/ufunclike.py", line 202, in isneginf
return nx.logical_and(nx.isinf(x), nx.signbit(x), out)
srun: error: sls-titan-0: task 0: Exited with exit code 2
The exception is thrown by numpy.isneginf(lp) in commands/score.py, line 209. Apparently lp, which should be floating point (logprob) is something else. Can you debug this by printing the value? For example, before the line in question, add this:
for lp in merged_logprobs:
print(lp)
print(numpy.isneginf(lp))
and see what is the last value printed. Alternatively, if you give me the data and instructions how to reproduce the error, I can debug it myself.
-8.192791
False
-2.4449062
False
-6.7236123
False
-6.914638
False
-7.526195
False
None
2018-04-08 20:15:46,030 exception_handler: An unexpected TypeError exception occurred: ufunc 'isinf' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''
Traceback will be written to debug log (enable with --log-level debug).
2018-04-08 20:15:46,030 exception_handler: Traceback:
2018-04-08 20:15:46,039 exception_handler: File "/data/sls/u/sameerk/anaconda3/envs/theano-lm/bin/theanolm", line 147, in <module>
main()
2018-04-08 20:15:46,039 exception_handler: File "/data/sls/u/sameerk/anaconda3/envs/theano-lm/bin/theanolm", line 88, in main
args.command_function(args)
2018-04-08 20:15:46,040 exception_handler: File "/data/sls/u/sameerk/anaconda3/envs/theano-lm/lib/python3.5/site-packages/theanolm/commands/score.py", line 122, in score
args.output_file, args.log_base, args.subwords, False)
2018-04-08 20:15:46,040 exception_handler: File "/data/sls/u/sameerk/anaconda3/envs/theano-lm/lib/python3.5/site-packages/theanolm/commands/score.py", line 210, in _score_text
print(numpy.isneginf(lp))
2018-04-08 20:15:46,040 exception_handler: File "/data/sls/u/sameerk/anaconda3/envs/theano-lm/lib/python3.5/site-packages/numpy/lib/ufunclike.py", line 34, in func
return f(x, out=out, **kwargs)
2018-04-08 20:15:46,040 exception_handler: File "/data/sls/u/sameerk/anaconda3/envs/theano-lm/lib/python3.5/site-packages/numpy/lib/ufunclike.py", line 202, in isneginf
return nx.logical_and(nx.isinf(x), nx.signbit(x), out)
srun: error: sls-titan-0: task 0: Exited with exit code 2
you can find the necessary files here:
http://people.csail.mit.edu/sameerk/for_theanolm/
check out the README.txt at the location for details. Let me know if you need more info.
Thanks for the help.
Thanks! I think you have to add a check here:
https://github.com/senarvi/theanolm/blob/master/theanolm/commands/score.py#L209
that the logporb is not None i.e. UNK when computing the number of zero-probabilities. Try this:
num_zeroprobs += sum((lp is not None) and numpy.isneginf(lp) for lp in merged_logprobs)
working now, thanks
Do you want to create a pull request so that I can merge the bug fix?
I commited the change to the develop branch. Can you pull that branch and check that it works for you?
hi,
when using the command
theanolm score model.h5 test-data.txt --output perplexity --exclude-unk
i get the following error:
without the option --exclude-unk, it gives me the ppl
Any comments?