`nbest_size` issue - Githubissues

Traceback (most recent call last):
  File "example.py", line 134, in <module>
    main()
  File "example.py", line 126, in main
    list_of_input_ids = tokenizer.list_of_string_to_list_of_cls_sep_token_ids([input_text])
  File "/home/yyoo/src/korner/data_utils/vocab_tokenizer.py", line 229, in list_of_string_to_list_of_cls_sep_token_ids
    X_token_batch = self.list_of_string_to_list_of_tokens(X_str_batch)
  File "/home/yyoo/src/korner/data_utils/vocab_tokenizer.py", line 188, in list_of_string_to_list_of_tokens
    X_token_batch = [self._split(X_str) for X_str in X_str_batch]
  File "/home/yyoo/src/korner/data_utils/vocab_tokenizer.py", line 188, in <listcomp>
    X_token_batch = [self._split(X_str) for X_str in X_str_batch]
  File "/home/yyoo/crf/lib/python3.8/site-packages/gluonnlp/data/transforms.py", line 560, in __call__
    return self._processor.SampleEncodeAsPieces(sample, self._nbest,
  File "/home/yyoo/crf/lib/python3.8/site-packages/sentencepiece/__init__.py", line 561, in SampleEncodeAsPieces
    return self.Encode(input=input, nbest_size=nbest_size, alpha=alpha,
  File "/home/yyoo/crf/lib/python3.8/site-packages/sentencepiece/__init__.py", line 503, in Encode
    raise RuntimeError(
RuntimeError: When enable_sampling is True, We must specify "nbest_size > 1" or "nbest_size = -1", and "alpha". "nbest_size" is enabled only on unigram mode ignored in BPE-dropout. when "nbest_size = -1" , this method samples from all candidates on the lattice instead of nbest segmentations.
ys7yoo / korner

`nbest_size` issue #5
