-
### 🚀 The feature, motivation and pitch
The current multilingual recipes are for LLAMA 2.
I would like to see LLAMA 3 multilingual recipes added.
Thank you.
### Alternatives
_No response_
##…
-
While EC2 is ok. On Amazon Lambda I get this error:
Traceback (most recent call last):
File "/var/task/tiktoken/registry.py", line 34, in _find_constructors
constructors = mod.ENCODING_CONSTR…
-
I'm attempting to train LLaMA-3 using Megatron-LM but have encountered an issue: LLaMA-3 utilizes Tiktoken for tokenization and doesn't provide a tokenizer.model file, which is required by Megatron-LM…
SDsly updated
2 weeks ago
-
I am trying to use Llamaindex with Plasmo framework on a Windows machine and tiktoken is throwing the above error.
`No transformers found for __node\_modules/tiktoken-node/dist/tiktoken-node.win32-…
-
Hi,
When I use:
pip install whisper-at
I get:
Defaulting to user installation because normal site-packages is not writeable
Collecting whisper-at
Downloading whisper_at-0.5-py3-none-any.wh…
-
The current `requirements.txt` describes the following packages, but I needed `tiktoken` to actually work.
```
openai
fastapi
black
isort
websockets
pydantic
langchain
uvicorn
jinja2
faiss-…
-
As I'm trying to get this configured I've run into the following issue after any `:Chat` call:
```
Error: post https://api.openai.com/v1/chat/completions - curl error exit_code=23
stderr={ "curl…
-
It seems that the tiktoken package is not linkable from Rust using Cargo's default registry.
Are there plans to publish the `tiktoken` crate? Is it published on another registry?
Thanks for your…
-
Hey, thanks for porting this over! I wanted to move to PHP to remove an extra dependency (docker server exposing Python TikToken over API). I decided to do a small benchmark and it seems that PHP vers…
-
I am using tiktoken in my project here: https://github.com/langfuse/langfuse/blob/250a5b03d2068774151757d8a0b33115a56243e2/src/features/ingest/lib/usage.ts#L6
As advised in the docs, i always free …