This PR adds a new argument to normalize_words() to optionally return scores for every candidate word outputted by the model. These scores are outputted by OpenNMT's Translator, so the changes only involve capturing this output.
Summary of changes
Prediction scores are stored in the fake_stream class during a translation run and are called from that class after the translations are complete
Output of call_onmt() is now a list of candidate tuples, where the first position in a tuple is the candidate word and the second is the prediction score
Small modification to _dict_filter() involves handling the candidate tuples
_normalize() now ends with an if/else check to determine whether the candidate words and their scores should be returned, or just the words
This PR adds a new argument to
normalize_words()
to optionally return scores for every candidate word outputted by the model. These scores are outputted by OpenNMT'sTranslator
, so the changes only involve capturing this output.Summary of changes
fake_stream
class during a translation run and are called from that class after the translations are completecall_onmt()
is now a list of candidate tuples, where the first position in a tuple is the candidate word and the second is the prediction score_dict_filter()
involves handling the candidate tuples_normalize()
now ends with anif/else
check to determine whether the candidate words and their scores should be returned, or just the words