Open CoinCheung opened 5 months ago
tokenizer collapse
No response
model_name = 'THUDM/chatglm3-6b-base' config = AutoConfig.from_pretrained(model_name, trust_remote_code=True, use_fast=False) tokenizer = AutoTokenizer.from_pretrained(model_name, config=config, trust_remote_code=True) print([tokenizer.decode(el) for el in [1833, 2893]])
- OS: - Python: - Transformers: - PyTorch: - CUDA Support (`python -c "import torch; print(torch.cuda.is_available())"`) :
On my platform, the output is same:
Is there an existing issue for this?
Current Behavior
tokenizer collapse
Expected Behavior
No response
Steps To Reproduce
model_name = 'THUDM/chatglm3-6b-base' config = AutoConfig.from_pretrained(model_name, trust_remote_code=True, use_fast=False) tokenizer = AutoTokenizer.from_pretrained(model_name, config=config, trust_remote_code=True) print([tokenizer.decode(el) for el in [1833, 2893]])
Environment
Anything else?
On my platform, the output is same: