Tiktoken package for openai script not working

azmathr commented 9 months ago

In my pthon script I have tiktoken enabled which essencially converts characters to openai tokens. Its a common module used in openai. Here is an associated code.

encoding = tiktoken.get_encoding("cl100k_base")

`def num_tokens_from_string(string: str, encoding_name: str) -> int: """Returns the number of tokens in a text string.""" encoding = tiktoken.get_encoding(encoding_name) num_tokens = len(encoding.encode(string)) return num_tokens

def get_text_until_limit(elements, limit): text = '' for element in elements: new_text = text + ' ' + element.get_text() if num_tokens_from_string(new_text, "cl100k_base") <= limit: text = new_text else: break return text ` Unfortunately this is not working through zappa. I am getting error

encoding = tiktoken.get_encoding("cl100k_base") File "/var/task/tiktoken/registry.py", line 64, in get_encoding _find_constructors() File "/var/task/tiktoken/registry.py", line 44, in _find_constructors raise ValueError( [1707923842317] [DEBUG] 2024-02-1

warmhandler error

souravjamwal77 commented 9 months ago

@azmathr Can you please post the full error trace?

azmathr commented 9 months ago

It was solved by deleting the pycache file in tiktoken and site package folder

Sent from Gmail Mobile

On Fri, Feb 23, 2024 at 3:40 PM Sourav Kumar @.***> wrote:

@azmathr https://github.com/azmathr Can you please post the full error trace?

— Reply to this email directly, view it on GitHub https://github.com/zappa/Zappa/issues/1306#issuecomment-1961028222, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABEK25NEBOGBFM2GX5AHKCLYVBRQDAVCNFSM6AAAAABDKGZKAKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRRGAZDQMRSGI . You are receiving this because you were mentioned.Message ID: @.***>

monkut commented 8 months ago

closing this issue as it was resolved by the op.

zappa / Zappa

Tiktoken package for openai script not working #1306