-
On a fresh container with minimal prereqs, first call results in:
`Traceback (most recent call last):
File "/opt/conda/lib/python3.10/site-packages/langchain/llms/openai.py", line 233, in get_nu…
-
I was playing with the tokenizer, and I noticed some missed merge opportunities.
```python
>>> tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")
>>> tokenizer(['\t', '\t\t',…
-
Hello! I don't remember if I'd shown you https://github.com/openai/tiktoken/blob/main/tiktoken/_educational.py , but consider stealing the token visualisation code from here in some form: https://gith…
-
Upon executing this:
charts = lida.visualize(summary=summary, goal=user_query, textgen_config=textgen_config)
I am getting error:
name 'tiktoken' is not defined
-
### Describe the issue
When running the [pgvector example](https://github.com/microsoft/autogen/blob/main/notebook/agentchat_pgvector_RetrieveChat.ipynb) I get the following error:
m:\One…
-
Looking through Llama3 changes, I see that "ignore_merges" was added as a property to support conversion from tiktoken models. Can a native HF tokenizer be trained using this property? It's not clear…
-
### Bug Description
I recently conduced a few experiments using RaptorPack and everything looks fine. The only fallback is that the token counter is not working for the RaptorPack so that I cannot ge…
-
### Describe the bug
`completion_tokens += 1` looks like a bad idea. Do we expect a chunk to always be 1 token? I don't think so.
### Steps to reproduce
_No response_
### Expected Behavior…
-
### 🚀 The feature, motivation and pitch
The current multilingual recipes are for LLAMA 2.
I would like to see LLAMA 3 multilingual recipes added.
Thank you.
### Alternatives
_No response_
##…
-
I'm attempting to train LLaMA-3 using Megatron-LM but have encountered an issue: LLaMA-3 utilizes Tiktoken for tokenization and doesn't provide a tokenizer.model file, which is required by Megatron-LM…
SDsly updated
2 weeks ago