tiktoken calculation error with special characters in devchat-api

devchat-ai / devchat

Automate your dev tasks with AI-powered scripts, from your IDE's chat panel.

https://www.devchat.ai

Apache License 2.0

351 stars 42 forks source link

tiktoken calculation error with special characters in devchat-api #215

Closed yangbobo2021 closed 9 months ago

yangbobo2021 commented 9 months ago

Description

When performing tiktoken calculations, an error is encountered with special characters. The devchat-api project also receives these special characters which leads to the error: