search

predibase / lorax

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs
https://loraexchange.ai
Apache License 2.0
2.18k stars 143 forks source link

Support custom tokenizer when loading a local model #151

Open yinjiaoyuan opened 10 months ago

yinjiaoyuan commented 10 months ago

Feature request

I have download the model, so I want to run it use local model, eht sample is: docker run --gpus all --shm-size 1g -p 8080:80 -v /data/model/:/data/ \ ghcr.io/predibase/lorax:latest --model-id /data/model/Qwen-14B-Chat

Motivation

I want to use the local model. Our computes don't allow to visit huggingface.co.

Your contribution

No.

yinjiaoyuan commented 10 months ago

I use the below code to save model: from transformers import AutoModelForCausalLM, AutoTokenizer from transformers.generation import GenerationConfig

model_dir = '/data/model/Qwen-14B-Chat-hf-save'

name = 'Qwen/Qwen-14B-Chat' tokenizer = AutoTokenizer.from_pretrained(name, trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained(name, device_map="auto", trust_remote_code=True, bf16=True).eval() tokenizer.save_pretrained(model_dir) model.save_pretrained(model_dir)

And run: docker run --gpus all --shm-size 1g -p 8080:80 -v /data/model/:/data/ \ ghcr.io/predibase/lorax:latest --model-id /data/Qwen-14B-Chat-hf-save

The error is: 2023-12-26T02:25:38.134822Z INFO lorax_launcher: Args { model_id: "/data/Qwen-14B-Chat-hf-save", adapter_id: "", source: "hub", adapter_source: "hub", revision: None, validation_workers: 2, sharded: None, num_shard: None, quantize: None, dtype: None, trust_remote_code: false, max_concurrent_requests: 128, max_best_of: 2, max_stop_sequences: 4, max_input_length: 1024, max_total_tokens: 2048, waiting_served_ratio: 1.2, max_batch_prefill_tokens: 4096, max_batch_total_tokens: None, max_waiting_tokens: 20, max_active_adapters: 128, adapter_cycle_time_s: 2, hostname: "dd28579e51fc", port: 80, shard_uds_path: "/tmp/lorax-server", master_addr: "localhost", master_port: 29500, huggingface_hub_cache: Some("/data"), weights_cache_override: None, disable_custom_kernels: false, cuda_memory_fraction: 1.0, json_output: false, otlp_endpoint: None, cors_allow_origin: [], watermark_gamma: None, watermark_delta: None, ngrok: false, ngrok_authtoken: None, ngrok_edge: None, env: false, download_only: false } 2023-12-26T02:25:38.134920Z INFO download: lorax_launcher: Starting download process. 2023-12-26T02:25:41.114273Z INFO lorax_launcher: cli.py:103 Files are already present on the host. Skipping download.

2023-12-26T02:25:41.537340Z INFO download: lorax_launcher: Successfully downloaded weights. 2023-12-26T02:25:41.537502Z INFO shard-manager: lorax_launcher: Starting shard rank=0 2023-12-26T02:25:44.896833Z ERROR lorax_launcher: server.py:233 Error when initializing model Traceback (most recent call last): File "/opt/conda/bin/lorax-server", line 8, in sys.exit(app()) File "/opt/conda/lib/python3.10/site-packages/typer/main.py", line 311, in call return get_command(self)(*args, kwargs) File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1157, in call return self.main(args, kwargs) File "/opt/conda/lib/python3.10/site-packages/typer/core.py", line 778, in main return _main( File "/opt/conda/lib/python3.10/site-packages/typer/core.py", line 216, in _main rv = self.invoke(ctx) File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, ctx.params) File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 783, in invoke return __callback(args, kwargs) File "/opt/conda/lib/python3.10/site-packages/typer/main.py", line 683, in wrapper return callback(*use_params) # type: ignore File "/opt/conda/lib/python3.10/site-packages/lorax_server/cli.py", line 84, in serve server.serve( File "/opt/conda/lib/python3.10/site-packages/lorax_server/server.py", line 277, in serve asyncio.run( File "/opt/conda/lib/python3.10/asyncio/runners.py", line 44, in run return loop.run_until_complete(main) File "/opt/conda/lib/python3.10/asyncio/base_events.py", line 636, in run_until_complete self.run_forever() File "/opt/conda/lib/python3.10/asyncio/base_events.py", line 603, in run_forever self._run_once() File "/opt/conda/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once handle._run() File "/opt/conda/lib/python3.10/asyncio/events.py", line 80, in _run self._context.run(self._callback, self._args)

File "/opt/conda/lib/python3.10/site-packages/lorax_server/server.py", line 229, in serve_inner model = get_model( File "/opt/conda/lib/python3.10/site-packages/lorax_server/models/init.py", line 321, in get_model return FlashQwen( File "/opt/conda/lib/python3.10/site-packages/lorax_server/models/flash_qwen.py", line 54, in init tokenizer = AutoTokenizer.from_pretrained( File "/opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 784, in from_pretrained raise ValueError( ValueError: Tokenizer class QWenTokenizer does not exist or is not currently imported.

2023-12-26T02:25:45.540361Z ERROR shard-manager: lorax_launcher: Shard complete standard error output:

Traceback (most recent call last):

File "/opt/conda/bin/lorax-server", line 8, in sys.exit(app())

File "/opt/conda/lib/python3.10/site-packages/lorax_server/cli.py", line 84, in serve server.serve(

File "/opt/conda/lib/python3.10/site-packages/lorax_server/server.py", line 277, in serve asyncio.run(

File "/opt/conda/lib/python3.10/asyncio/runners.py", line 44, in run return loop.run_until_complete(main)

File "/opt/conda/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete return future.result()

File "/opt/conda/lib/python3.10/site-packages/lorax_server/server.py", line 229, in serve_inner model = get_model(

File "/opt/conda/lib/python3.10/site-packages/lorax_server/models/init.py", line 321, in get_model return FlashQwen(

File "/opt/conda/lib/python3.10/site-packages/lorax_server/models/flash_qwen.py", line 54, in init tokenizer = AutoTokenizer.from_pretrained(

File "/opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 784, in from_pretrained raise ValueError(

ValueError: Tokenizer class QWenTokenizer does not exist or is not currently imported. rank=0 2023-12-26T02:25:45.639763Z ERROR lorax_launcher: Shard 0 failed to start 2023-12-26T02:25:45.639783Z INFO lorax_launcher: Shutting down shards Error: ShardCannotStart

The dir /data/model/Qwen-14B-Chat-hf-save: total 27673116 drwxr-xr-x 2 root root 4096 Dec 25 19:14 ./ drwxr-xr-x 3 root root 4096 Dec 26 10:25 ../ -rw-r--r-- 1 root root 1110 Dec 25 19:13 config.json -rw-r--r-- 1 root root 250 Dec 25 19:13 generation_config.json -rw-r--r-- 1 root root 4919444336 Dec 25 19:13 model-00001-of-00006.safetensors -rw-r--r-- 1 root root 4991627864 Dec 25 19:13 model-00002-of-00006.safetensors -rw-r--r-- 1 root root 4886749824 Dec 25 19:13 model-00003-of-00006.safetensors -rw-r--r-- 1 root root 4903809664 Dec 25 19:14 model-00004-of-00006.safetensors -rw-r--r-- 1 root root 4903820016 Dec 25 19:14 model-00005-of-00006.safetensors -rw-r--r-- 1 root root 3729165312 Dec 25 19:14 model-00006-of-00006.safetensors -rw-r--r-- 1 root root 24387 Dec 25 19:14 model.safetensors.index.json -rw-r--r-- 1 root root 2561218 Dec 25 19:13 qwen.tiktoken -rw-r--r-- 1 root root 3 Dec 25 19:13 special_tokens_map.json -rw-r--r-- 1 root root 261 Dec 25 19:13 tokenizer_config.json

yinjiaoyuan commented 10 months ago

Or support model from www.modelscope.cn like vllm？

tgaddair commented 10 months ago

Hey @yinjiaoyuan, apologies for the delayed response, I've been out on holiday this week.

To answer your first question, it looks like the current issue you're seeing is due to the model using a third party tokenizer (not natively part of HF):

ValueError: Tokenizer class QWenTokenizer does not exist or is not currently imported.

You can resolve this issue by running LoRAX with --trust-remote-code so it downloads the tokenizer from the model repo.

Let me know if that resolves your issue.

I'd be happy to add support for ModelScope, though I would need to explore it to understand the API a bit better (sounds like vllm would be a good reference for that). Contributions also welcome if you'd be interested in adding it.

yinjiaoyuan commented 10 months ago

But my linux server can't visit the site "https://huggingface.co/" in China.

tgaddair commented 10 months ago

Hey @yinjiaoyuan, you should be able to save the HF tokenizer to the same directory you saved the model weights to, which will allow LoRAX to read from the cache instead of going over the internet. For example, here are the contents of my local directory cache:

ls /data/models--Qwen--Qwen-7B/snapshots/ffe04dd57f85293043ba999a2c0daa788d6182e9/
config.json                       model-00003-of-00008.safetensors  model-00006-of-00008.safetensors  qwen.tiktoken
model-00001-of-00008.safetensors  model-00004-of-00008.safetensors  model-00007-of-00008.safetensors  tokenization_qwen.py
model-00002-of-00008.safetensors  model-00005-of-00008.safetensors  model-00008-of-00008.safetensors  tokenizer_config.json

You can see there's a file called tokenization_qwen.py that contains the tokenizer implementation. If you can write out this file locally, you should be able to avoid hitting the HF servers.

yinjiaoyuan commented 10 months ago

Hey @yinjiaoyuan, you should be able to save the HF tokenizer to the same directory you saved the model weights to, which will allow LoRAX to read from the cache instead of going over the internet. For example, here are the contents of my local directory cache:

ls /data/models--Qwen--Qwen-7B/snapshots/ffe04dd57f85293043ba999a2c0daa788d6182e9/
config.json                       model-00003-of-00008.safetensors  model-00006-of-00008.safetensors  qwen.tiktoken
model-00001-of-00008.safetensors  model-00004-of-00008.safetensors  model-00007-of-00008.safetensors  tokenization_qwen.py
model-00002-of-00008.safetensors  model-00005-of-00008.safetensors  model-00008-of-00008.safetensors  tokenizer_config.json

You can see there's a file called tokenization_qwen.py that contains the tokenizer implementation. If you can write out this file locally, you should be able to avoid hitting the HF servers.

Yes, I dowload the hf model on the foreign host and scp the model files to the host in the China, but I run lorax and input --model-id /xxx/yyy/data/models--Qwen--Qwen-7B-Chat/snapshots/218aa3240fd5a5d1e80bb6c47d5d774361913706/, /data/models--Qwen--Qwen-7B-Chat/snapshots/218aa3240fd5a5d1e80bb6c47d5d774361913706/ is the dir of the model fies, but NOT ok, the error info is: 2024-01-08T03:11:04.139756Z INFO lorax_launcher: Args { model_id: "/data/models--Qwen--Qwen-7B-Chat/snapshots/218aa3240fd5a5d1e80bb6c47d5d774361913706/", adapter_id: "", source: "hub", adapter_source: "hub", revision: None, validation_workers: 2, sharded: None, num_shard: None, quantize: None, dtype: None, trust_remote_code: false, max_concurrent_requests: 128, max_best_of: 2, max_stop_sequences: 4, max_input_length: 1024, max_total_tokens: 2048, waiting_served_ratio: 1.2, max_batch_prefill_tokens: 4096, max_batch_total_tokens: None, max_waiting_tokens: 20, max_active_adapters: 128, adapter_cycle_time_s: 2, hostname: "f06b11000cff", port: 80, shard_uds_path: "/tmp/lorax-server", master_addr: "localhost", master_port: 29500, huggingface_hub_cache: Some("/data"), weights_cache_override: None, disable_custom_kernels: false, cuda_memory_fraction: 1.0, json_output: false, otlp_endpoint: None, cors_allow_origin: [], watermark_gamma: None, watermark_delta: None, ngrok: false, ngrok_authtoken: None, ngrok_edge: None, env: false, download_only: false } 2024-01-08T03:11:04.139862Z INFO download: lorax_launcher: Starting download process. 2024-01-08T03:11:07.000076Z INFO lorax_launcher: cli.py:103 Files are already present on the host. Skipping download.

2024-01-08T03:11:07.442269Z INFO download: lorax_launcher: Successfully downloaded weights. 2024-01-08T03:11:07.442447Z INFO shard-manager: lorax_launcher: Starting shard rank=0 2024-01-08T03:11:11.104509Z ERROR lorax_launcher: server.py:233 Error when initializing model Traceback (most recent call last): File "/opt/conda/bin/lorax-server", line 8, in sys.exit(app()) File "/opt/conda/lib/python3.10/site-packages/typer/main.py", line 311, in call return get_command(self)(*args, kwargs) File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1157, in call return self.main(args, kwargs) File "/opt/conda/lib/python3.10/site-packages/typer/core.py", line 778, in main return _main( File "/opt/conda/lib/python3.10/site-packages/typer/core.py", line 216, in _main rv = self.invoke(ctx) File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, ctx.params) File "/opt/conda/lib/python3.10/site-packages/click/core.py", line 783, in invoke return __callback(args, kwargs) File "/opt/conda/lib/python3.10/site-packages/typer/main.py", line 683, in wrapper return callback(*use_params) # type: ignore File "/opt/conda/lib/python3.10/site-packages/lorax_server/cli.py", line 84, in serve server.serve( File "/opt/conda/lib/python3.10/site-packages/lorax_server/server.py", line 277, in serve asyncio.run( File "/opt/conda/lib/python3.10/asyncio/runners.py", line 44, in run return loop.run_until_complete(main) File "/opt/conda/lib/python3.10/asyncio/base_events.py", line 636, in run_until_complete self.run_forever() File "/opt/conda/lib/python3.10/asyncio/base_events.py", line 603, in run_forever self._run_once() File "/opt/conda/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once handle._run() File "/opt/conda/lib/python3.10/asyncio/events.py", line 80, in _run self._context.run(self._callback, self._args)

File "/opt/conda/lib/python3.10/site-packages/lorax_server/server.py", line 229, in serve_inner model = get_model( File "/opt/conda/lib/python3.10/site-packages/lorax_server/models/init.py", line 321, in get_model return FlashQwen( File "/opt/conda/lib/python3.10/site-packages/lorax_server/models/flash_qwen.py", line 54, in init tokenizer = AutoTokenizer.from_pretrained( File "/opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 784, in from_pretrained raise ValueError( ValueError: Tokenizer class QWenTokenizer does not exist or is not currently imported.

2024-01-08T03:11:11.645631Z ERROR shard-manager: lorax_launcher: Shard complete standard error output:

Traceback (most recent call last):

File "/opt/conda/bin/lorax-server", line 8, in sys.exit(app())

File "/opt/conda/lib/python3.10/site-packages/lorax_server/cli.py", line 84, in serve server.serve(

File "/opt/conda/lib/python3.10/site-packages/lorax_server/server.py", line 277, in serve asyncio.run(

File "/opt/conda/lib/python3.10/asyncio/runners.py", line 44, in run return loop.run_until_complete(main)

File "/opt/conda/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete return future.result()

File "/opt/conda/lib/python3.10/site-packages/lorax_server/server.py", line 229, in serve_inner model = get_model(

File "/opt/conda/lib/python3.10/site-packages/lorax_server/models/init.py", line 321, in get_model return FlashQwen(

File "/opt/conda/lib/python3.10/site-packages/lorax_server/models/flash_qwen.py", line 54, in init tokenizer = AutoTokenizer.from_pretrained(

File "/opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 784, in from_pretrained raise ValueError(

ValueError: Tokenizer class QWenTokenizer does not exist or is not currently imported. rank=0 2024-01-08T03:11:11.745313Z ERROR lorax_launcher: Shard 0 failed to start 2024-01-08T03:11:11.745333Z INFO lorax_launcher: Shutting down shards Error: ShardCannotStart

tgaddair commented 10 months ago

Hey @yinjiaoyuan, sorry about the back and forth. That definitely looks like a bug, then. We should be able to obtain the local tokenizer without going to HF. I'll see if I can repro the error.

Can you run ls /data/models--Qwen--Qwen-7B-Chat/snapshots/218aa3240fd5a5d1e80bb6c47d5d774361913706/ and paste the results here to confirm that tokenization_qwen.py is in the directory?

yinjiaoyuan commented 10 months ago

Hey @yinjiaoyuan, sorry about the back and forth. That definitely looks like a bug, then. We should be able to obtain the local tokenizer without going to HF. I'll see if I can repro the error.

Can you run ls /data/models--Qwen--Qwen-7B-Chat/snapshots/218aa3240fd5a5d1e80bb6c47d5d774361913706/ and paste the results here to confirm that tokenization_qwen.py is in the directory?

ls /data/models--Qwen--Qwen-7B-Chat/snapshots/218aa3240fd5a5d1e80bb6c47d5d774361913706/: total 15083436 drwxr-xr-x 2 root root 4096 Jan 3 11:05 ./ drwxr-xr-x 3 root root 4096 Jan 3 10:40 ../ -rw-r--r-- 1 root root 911 Jan 3 10:40 config.json -rw-r--r-- 1 root root 2345 Jan 3 10:40 configuration_qwen.py -rw-r--r-- 1 root root 1924 Jan 3 10:40 cpp_kernels.py -rw-r--r-- 1 root root 273 Jan 3 11:05 generation_config.json -rw-r--r-- 1 root root 1964066488 Jan 3 10:43 model-00001-of-00008.safetensors -rw-r--r-- 1 root root 2023960808 Jan 3 10:47 model-00002-of-00008.safetensors -rw-r--r-- 1 root root 2023960816 Jan 3 10:51 model-00003-of-00008.safetensors -rw-r--r-- 1 root root 2023960848 Jan 3 10:54 model-00004-of-00008.safetensors -rw-r--r-- 1 root root 2023960848 Jan 3 10:57 model-00005-of-00008.safetensors -rw-r--r-- 1 root root 2023960848 Jan 3 11:00 model-00006-of-00008.safetensors -rw-r--r-- 1 root root 2023960848 Jan 3 11:03 model-00007-of-00008.safetensors -rw-r--r-- 1 root root 1334845784 Jan 3 11:05 model-00008-of-00008.safetensors -rw-r--r-- 1 root root 55563 Jan 3 10:40 modeling_qwen.py -rw-r--r-- 1 root root 19547 Jan 3 10:40 model.safetensors.index.json -rw-r--r-- 1 root root 14604 Jan 3 10:40 qwen_generation_utils.py -rw-r--r-- 1 root root 2561218 Jan 3 10:40 qwen.tiktoken -rw-r--r-- 1 root root 9618 Jan 3 10:40 tokenization_qwen.py -rw-r--r-- 1 root root 174 Jan 3 10:40 tokenizer_config.json

And cat /data/models--Qwen--Qwen-7B-Chat/snapshots/218aa3240fd5a5d1e80bb6c47d5d774361913706/tokenization_qwen.py:

Copyright (c) Alibaba Cloud.

#

This source code is licensed under the license found in the

LICENSE file in the root directory of this source tree.

"""Tokenization classes for QWen."""

import base64 import logging import os import unicodedata from typing import Collection, Dict, List, Set, Tuple, Union

import tiktoken from transformers import PreTrainedTokenizer, AddedToken

logger = logging.getLogger(name)

VOCAB_FILES_NAMES = {"vocab_file": "qwen.tiktoken"}

PAT_STR = r"""(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}| ?[^\s\p{L}\p{N}]+[\r\n]|\s[\r\n]+|\s+(?!\S)|\s+""" ENDOFTEXT = "<|endoftext|>" IMSTART = "<|im_start|>" IMEND = "<|im_end|>"

as the default behavior is changed to allow special tokens in

regular texts, the surface forms of special tokens need to be

as different as possible to minimize the impact

EXTRAS = tuple((f"<|extra_{i}|>" for i in range(205)))

changed to use actual index to avoid misconfiguration with vocabulary expansion

SPECIAL_START_ID = 151643 SPECIAL_TOKENS = tuple( enumerate( ( ( ENDOFTEXT, IMSTART, IMEND, )

def _load_tiktoken_bpe(tiktoken_bpe_file: str) -> Dict[bytes, int]: with open(tiktoken_bpe_file, "rb") as f: contents = f.read() return { base64.b64decode(token): int(rank) for token, rank in (line.split() for line in contents.splitlines() if line) }

class QWenTokenizer(PreTrainedTokenizer): """QWen tokenizer."""

vocab_files_names = VOCAB_FILES_NAMES

def __init__(
    self,
    vocab_file,
    errors="replace",
    extra_vocab_file=None,
    **kwargs,
):
    super().__init__(**kwargs)

    # how to handle errors in decoding UTF-8 byte sequences
    # use ignore if you are in streaming inference
    self.errors = errors  

    self.mergeable_ranks = _load_tiktoken_bpe(vocab_file)  # type: Dict[bytes, int]
    self.special_tokens = {
        token: index
        for index, token in SPECIAL_TOKENS
    }

    # try load extra vocab from file
    if extra_vocab_file is not None:
        used_ids = set(self.mergeable_ranks.values()) | set(self.special_tokens.values())
        extra_mergeable_ranks = _load_tiktoken_bpe(extra_vocab_file)
        for token, index in extra_mergeable_ranks.items():
            if token in self.mergeable_ranks:
                logger.info(f"extra token {token} exists, skipping")
                continue
            if index in used_ids:
                logger.info(f'the index {index} for extra token {token} exists, skipping')
                continue
            self.mergeable_ranks[token] = index
        # the index may be sparse after this, but don't worry tiktoken.Encoding will handle this

    enc = tiktoken.Encoding(
        "Qwen",
        pat_str=PAT_STR,
        mergeable_ranks=self.mergeable_ranks,
        special_tokens=self.special_tokens,
    )
    assert (
        len(self.mergeable_ranks) + len(self.special_tokens) == enc.n_vocab
    ), f"{len(self.mergeable_ranks) + len(self.special_tokens)} != {enc.n_vocab} in encoding"

    self.decoder = {
        v: k for k, v in self.mergeable_ranks.items()
    }  # type: dict[int, bytes|str]
    self.decoder.update({v: k for k, v in self.special_tokens.items()})

    self.tokenizer = enc  # type: tiktoken.Encoding

    self.eod_id = self.tokenizer.eot_token
    self.im_start_id = self.special_tokens[IMSTART]
    self.im_end_id = self.special_tokens[IMEND]

def __getstate__(self):
    # for pickle lovers
    state = self.__dict__.copy()
    del state["tokenizer"]
    return state

def __setstate__(self, state):
    # tokenizer is not python native; don't pass it; rebuild it
    self.__dict__.update(state)
    enc = tiktoken.Encoding(
        "Qwen",
        pat_str=PAT_STR,
        mergeable_ranks=self.mergeable_ranks,
        special_tokens=self.special_tokens,
    )
    self.tokenizer = enc

def __len__(self) -> int:
    return self.tokenizer.n_vocab

def get_vocab(self) -> Dict[bytes, int]:
    return self.mergeable_ranks

def convert_tokens_to_ids(
    self, tokens: Union[bytes, str, List[Union[bytes, str]]]
) -> List[int]:
    ids = []
    if isinstance(tokens, (str, bytes)):
        if tokens in self.special_tokens:
            return self.special_tokens[tokens]
        else:
            return self.mergeable_ranks.get(tokens)
    for token in tokens:
        if token in self.special_tokens:
            ids.append(self.special_tokens[token])
        else:
            ids.append(self.mergeable_ranks.get(token))
    return ids

def _add_tokens(
    self,
    new_tokens: Union[List[str], List[AddedToken]],
    special_tokens: bool = False,
) -> int:
    if not special_tokens and new_tokens:
        raise ValueError("Adding regular tokens is not supported")
    for token in new_tokens:
        surface_form = token.content if isinstance(token, AddedToken) else token
        if surface_form not in SPECIAL_TOKENS_SET:
            raise ValueError("Adding unknown special tokens is not supported")
    return 0

def save_vocabulary(self, save_directory: str, **kwargs) -> Tuple[str]:
    """
    Save only the vocabulary of the tokenizer (vocabulary).

    Returns:
        `Tuple(str)`: Paths to the files saved.
    """
    file_path = os.path.join(save_directory, "qwen.tiktoken")
    with open(file_path, "w", encoding="utf8") as w:
        for k, v in self.mergeable_ranks.items():
            line = base64.b64encode(k).decode("utf8") + " " + str(v) + "\n"
            w.write(line)
    return (file_path,)

def tokenize(
    self,
    text: str,
    allowed_special: Union[Set, str] = "all",
    disallowed_special: Union[Collection, str] = (),
    **kwargs,
) -> List[Union[bytes, str]]:
    """
    Converts a string in a sequence of tokens.

    Args:
        text (`str`):
            The sequence to be encoded.
        allowed_special (`Literal["all"]` or `set`):
            The surface forms of the tokens to be encoded as special tokens in regular texts.
            Default to "all".
        disallowed_special (`Literal["all"]` or `Collection`):
            The surface forms of the tokens that should not be in regular texts and trigger errors.
            Default to an empty tuple.

        kwargs (additional keyword arguments, *optional*):
            Will be passed to the underlying model specific encode method.

    Returns:
        `List[bytes|str]`: The list of tokens.
    """
    tokens = []
    text = unicodedata.normalize("NFC", text)

    # this implementation takes a detour: text -> token id -> token surface forms
    for t in self.tokenizer.encode(
        text, allowed_special=allowed_special, disallowed_special=disallowed_special
    ):
        tokens.append(self.decoder[t])
    return tokens

def convert_tokens_to_string(self, tokens: List[Union[bytes, str]]) -> str:
    """
    Converts a sequence of tokens in a single string.
    """
    text = ""
    temp = b""
    for t in tokens:
        if isinstance(t, str):
            if temp:
                text += temp.decode("utf-8", errors=self.errors)
                temp = b""
            text += t
        elif isinstance(t, bytes):
            temp += t
        else:
            raise TypeError("token should only be of type types or str")
    if temp:
        text += temp.decode("utf-8", errors=self.errors)
    return text

@property
def vocab_size(self):
    return self.tokenizer.n_vocab

def _convert_id_to_token(self, index: int) -> Union[bytes, str]:
    """Converts an id to a token, special tokens included"""
    if index in self.decoder:
        return self.decoder[index]
    raise ValueError("unknown ids")

def _convert_token_to_id(self, token: Union[bytes, str]) -> int:
    """Converts a token to an id using the vocab, special tokens included"""
    if token in self.special_tokens:
        return self.special_tokens[token]
    if token in self.mergeable_ranks:
        return self.mergeable_ranks[token]
    raise ValueError("unknown token")

def _tokenize(self, text: str, **kwargs):
    """
    Converts a string in a sequence of tokens (string), using the tokenizer. Split in words for word-based
    vocabulary or sub-words for sub-word-based vocabularies (BPE/SentencePieces/WordPieces).

    Do NOT take care of added tokens.
    """
    raise NotImplementedError

def _decode(
    self,
    token_ids: Union[int, List[int]],
    skip_special_tokens: bool = False,
    errors: str = None,
    **kwargs,
) -> str:
    if isinstance(token_ids, int):
        token_ids = [token_ids]
    if skip_special_tokens:
        token_ids = [i for i in token_ids if i < self.eod_id]
    return self.tokenizer.decode(token_ids, errors=errors or self.errors)