search

openai / tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.
MIT License
11.76k stars 801 forks source link

fix pycache folder being added instead of encoding info files #152

Open dheerajiiitv opened 1 year ago

dheerajiiitv commented 1 year ago

Fix #61

CC: @hauntsaninja

dheerajiiitv commented 1 year ago

@hauntsaninja please review

rishabh-sagar-20 commented 8 months ago

Any update on this @hauntsaninja?

AlexanderCollins commented 7 months ago

@dheerajiiitv did you test this on aws lambda? I tried your changes in lambda with no success, same issue persists.

dheerajiiitv commented 7 months ago

@AlexanderCollins I did not test it on AWS lambda. Can you check and debug the values in the filter list? And also how did you build the package with my changes?

azmathr commented 7 months ago

@dheerajiiitv I get error in lambda while deploying through zappa

  File "/var/task/summaryserver.py", line 13, in <module>
    encoding = tiktoken.get_encoding("cl100k_base")
  File "/var/task/tiktoken/registry.py", line 64, in get_encoding
    _find_constructors()
  File "/var/task/tiktoken/registry.py", line 44, in _find_constructors
mpinta commented 6 months ago

Hey! Although this is not a solution for tiktoken, you can work around this error by adding the following to your zappa_settings.json file:

"exclude": [
    "pycache",
    "__pycache__"
]

Successfully tested on AWS Lambda.

azmathr commented 6 months ago

Thanks but that gives this error: Runtime.ImportModuleError: Unable to import module 'handler': No module named 'werkzeug' and Runtime.ImportModuleError: Unable to import module 'handler': attempted relative import with no known parent package

azmathr commented 6 months ago

I deleted pycache file from seite packages main folder and tiktoken folders and it worked!

rishabh-sagar-20 commented 5 months ago

Hey @hauntsaninja, is there any update on this? When are we merging this?