abdeladim-s / pyllamacpp

Python bindings for llama.cpp
https://abdeladim-s.github.io/pyllamacpp/
MIT License
62 stars 21 forks source link

Wrong description of detokenize() parameter 'tokens' #17

Closed RostF1rst closed 1 year ago

RostF1rst commented 1 year ago
def detokenize(self, tokens: list):
        """
        Returns a list of tokens for the text <- wrong description
        :param text: text to be tokenized <- wrong description
        :return: A string representing the text extracted from the tokens
        """
        return pp.llama_tokens_to_str(self._ctx, tokens)
abdeladim-s commented 1 year ago

Thanks @RostF1rst for pointing that out.

This function was introduced in a PR, so probably the developer was doing copy/paste from the tokenize function. I will try to fix that in the next release.