test sentinels and only append the token if not

mead-ml / mead-baseline

Deep-Learning Model Exploration and Development for NLP

Apache License 2.0

243 stars 73 forks source link

test sentinels and only append the token if not #884

Closed dpressel closed 2 years ago

dpressel commented 2 years ago

the previous approach meant that extra tokens were generated at the end which makes it unsuitable for most metrics, as the extra token is not present in the ref