-
```python
>>> tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")
>>> tokenizer("hello !")
{'input_ids': [128000, 15339, 758], 'attention_mask': [1, 1, 1]}
>>> tokenizer.decode…
-
python 3.10
powershell
管理员权限
debug my program, here is powershell output log of my python program:
晚上好 From: wxid_(privcy mask) To: wxid_(privcy mask) 'type': 1
ask:晚上好
Exception in threa…
-
Transform int4 cannot find `transformers_stream_generator einops tiktoken`
```bash
pip install transformers_stream_generator einops tiktoken
```
-
**Bug description**
1. platform
root@qingyi:~/MetaGPT/config# lsb_release -a
LSB Version: core-11.1.0ubuntu4-noarch:security-11.1.0ubuntu4-noarch
Distributor ID: Ubuntu
Description: …
-
### Details
在 src/pages/api/generate.ts 中加上和 src/components/Generator.tsx 中一样的裁剪 messages 的逻辑:
但是注意:服务端用不了 tiktoken 库,只能用 tiktoken-js 库,他们应该有类似的 interface
Checklist
- [X] Create `src/utils/tik…
-
**The bug**
I'm trying to run llaam-2-7b-chat-hf with togtherAI client. But I'm getting following error from tokenizer.
```
Exception: The tokenizer provided to the engine follows a non-ChatML fo…
-
https://github.com/openai/tiktoken.
in fact,its file "lib.rs" is easy to be compiled to wasm.
so it can be supported?
-
While installing using bun, getting the following error:
```
❯ bun i @anysphere/priompt
[0.04ms] ".env"
bun add v1.0.18 (36c316a2)
error: @anysphere/tikto…
-
I need to be able to call tiktoken in an environment where outbound traffic needs to be routed through a specific proxy. This is not possible with the current implementation of Tiktoken. One solution …
-
Does anybody saw this behavior?
What could be the reason?
What are my alternatives?
I'm in western Europe and https://openai.com is available
![image](https://github.com/openai/tiktoken/assets…