Can we get a `GeneratorCT2fromHfHub.generate_tokens`? 🚀

[JoaoLages]

Hi there! Thank you so much for your awesome work in this package and the open-source contributions 💪

I'm interested in having access to the native ctranslate2.Generator.generate_tokens method 👀

I'm using this code snippet:

from hf_hub_ctranslate2 import GeneratorCT2fromHfHub

model = GeneratorCT2fromHfHub(
    model_name_or_path="michaelfeil/ct2fast-Llama-2-7b-chat-hf",
    device="cuda",
    compute_type="int8_float16",
)
list(model.model.generate_tokens(["What is love?"]))

but the output is garbage 🚮

[GenerationStepResult(step=0, batch_id=0, token_id=7228, token='PA', log_prob=None, is_last=False),
 GenerationStepResult(step=1, batch_id=0, token_id=7228, token='PA', log_prob=None, is_last=False),
 GenerationStepResult(step=2, batch_id=0, token_id=7228, token='PA', log_prob=None, is_last=False),
 GenerationStepResult(step=3, batch_id=0, token_id=7228, token='PA', log_prob=None, is_last=False),
 GenerationStepResult(step=4, batch_id=0, token_id=7228, token='PA', log_prob=None, is_last=False),
 GenerationStepResult(step=5, batch_id=0, token_id=7228, token='PA', log_prob=None, is_last=False),
 GenerationStepResult(step=6, batch_id=0, token_id=7228, token='PA', log_prob=None, is_last=False),
 ...
 ]

What am I doing wrong here? 🤔

[michaelfeil]

Hi Joao, Great idea! If I find some spare time, I’ll try to add it!

In the meantime, you might want to try to tokenize with AutoTokenizer or Sentencpiece . Also try just to input a List[int] of token_ids?

[JoaoLages]

Just got it to work! It has to be a list of the tokens in str!

for x in model.model.generate_tokens(model.tokenizer.tokenize("What is love?")):
    pass