Use the value of the use_default_badwordids setting to configure bad_words_ids. Also add square brackets to bad_words_ids if use_default_badwordids is True. Fix an issue with attempting to use the tokenizer too early, and fix an exception populating Lua bridge data when zero tokens are generated, which can now happen if use_default_badwordids is False and the first token generated is EOS. Eliminate the need for trim_count by moving the EOS early-out up.
Use the value of the
use_default_badwordidssetting to configurebad_words_ids. Also add square brackets tobad_words_idsifuse_default_badwordidsis True. Fix an issue with attempting to use the tokenizer too early, and fix an exception populating Lua bridge data when zero tokens are generated, which can now happen ifuse_default_badwordidsis False and the first token generated is EOS. Eliminate the need fortrim_countby moving the EOS early-out up.