-
I'm using `js-tiktoken` and haven't been able to upgrade since version 1.0.7. It turns out that there are no ranks files in the latest version of js-tiktoken.
Is the removal of the ranks files inte…
-
自己本地修改master分支后构建镜像 docker build -t . --build-arg name=app --build-arg proxy=taobao 报错
=> ERROR [runner 13/17] COPY --from=mainDeps /app/node_modules/tiktoken ./node_modules/tiktoken …
-
**Describe the bug**
Cannot load such file -- tiktoken_ruby
File: `https://github.com/patterns-ai-core/langchainrb/blob/main/lib/langchain/utils/token_length/openai_validator.rb` has a require for…
-
Hi, I tried to use the GPT2 tokenizer of HF and TikToken, but I found TikToken is faster than HF. Could you explain why this might happen?
-
_Originally posted by @axvx in https://github.com/chaquo/chaquopy/issues/658#issuecomment-1480499875_
Hey why I see the following error:
Failed to install tiktoken from https://files…
-
I noticed our tests were much slower when switching from https://github.com/tiktoken-go/tokenizer to this library. It seems to be because `tiktoken.EncodingForModel` is very slow (0.3s on my machine).…
-
https://github.com/joaomdmoura/crewAI/blob/615ac7f297070950e140f0ac529e01f96fc02c64/src/crewai/utilities/token_counter_callback.py#L46
I am using the ChatOpenAI model from LangChain as my manager l…
-
-
I am trying to make mlc to support Qwen, but the model's tokenizer use tiktoken developed by openai, which is not supported by this repo now? So is there any plan for this feature?
-
GPT-4o support is not working properly.
Reason: tiktoken encoding needs to be updated for GPT-4o.
Currently encoding = tiktoken.get_encoding("gpt-3.5-turbo") is used (openai_helper.py row 656).
…