Research whether `extract_for_token_limit` needs to be updated to support 1106 models.

[leohpark]

Hello from linkedin!

I noticed the following function appears to assume the user will use one of ["gpt-4", "gpt-4-32k", "or gpt-3.5-turbo-16k"], and otherwise make some assumptions about model context limits (defaults to gpt-4), and returns a portion of document[] as a function of remaining_tokens. Assuming people are using the 1106 models, is this function still doing what is intended?

    def extract_for_token_limit(self, document, questions):
        encoding = tiktoken.encoding_for_model(self.model)
        question_string = ",".join(questions)
        questions_count = len(encoding.encode(question_string))
        user_prompt_count = len(encoding.encode(self.usecase_enhancement_user_prompt))
        system_prompt_count = len(encoding.encode(self.usecase_enhancement_system_prompt))
        extra_count = len(encoding.encode("'role', 'system', 'content', 'role', 'user', 'content'"))
        token_limit = 8192

        if "16k" in self.model:
            token_limit = 16384
        elif "32k" in self.model:
            token_limit = 32768

        remaining_tokens = token_limit - (questions_count + user_prompt_count + system_prompt_count + extra_count)
        document_encoding = encoding.encode(document)
        if len(encoding.encode(document)) <= remaining_tokens:
            return document

        #return encoding.decode(document_encoding[:remaining_tokens])
        return document[:remaining_tokens*3]

additionally, I would suggest using a proper text splitter tool in order to return a more precise slice of the document based on actual token count. There are examples at both extremes where tokens-per-character is not near the assumed values of 3 and 4.

[dgarnitz]

Hey thanks for commenting. To answer a few of your concerns:

• this is a POC right now thats really only designed to work with vanilla GPT-4. It does not perform well even with 3.5 turbo and we have not tested it against other llms (like PALM). 1106 models are for now out of scope.
• it also will not do any kind of sliding window extraction, so if you are for example using the 32K token limit model and pass in a 60K token document, it will not extract enhancement info from that remaining 28K of tokens
• this expects the user to have their own chunks already
• I totally get what you are saying about tokens. Hardcoding the slices is hacky. But I ran into a bug where I was using tiktoken to encode, count the number of tokens, slide the token array, then decode back and a bug was occurring where extra characters were getting inserted that caused the chat completion API to give wildly wrong answers. So I hardcoded it for now

[leohpark]

thanks for the response, makes sense. I'm pretty sure I've run into tiktoken bugs too, and they nearly drove me mad.

[dgarnitz]

It seems like there is not any pressing need on this so I am closing it out