using docker got errors

goldenquant commented 3 months ago

从 https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Instruct 下载了gguf 根据 Docker.md 启动docker，命令：

docker run --gpus all -v /data/models_dir/gguf/:/models -p 10002:10002 approachingai/ktransformers:0.1.1 --port 10002 --gguf_path /models/DeepSeek-V2-Coder --model_path /models/DeepSeek-V2-Coder --web True

报如下错误

Traceback (most recent call last):
  File "/opt/conda/bin/ktransformers", line 8, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/main.py", line 134, in main
    create_interface(config=cfg, default_args=default_args)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/utils/create_interface.py", line 27, in create_interface
    GlobalInterface.interface = BackendInterface(default_args)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 20, in __init__
    self.tokenizer = AutoTokenizer.from_pretrained(args.model_dir,device = args.device)
  File "/opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 853, in from_pretrained
    config = AutoConfig.from_pretrained(
  File "/opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 972, in from_pretrained
    config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/transformers/configuration_utils.py", line 632, in get_config_dict
    config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/transformers/configuration_utils.py", line 689, in _get_config_dict
    resolved_config_file = cached_file(
  File "/opt/conda/lib/python3.10/site-packages/transformers/utils/hub.py", line 373, in cached_file
    raise EnvironmentError(
OSError: /models/DeepSeek-V2-Coder does not appear to have a file named config.json. Checkout 'https://huggingface.co//models/DeepSeek-V2-Coder/tree/None' for available files.

去这个链接下面下载了config.json后，报错如下：

Traceback (most recent call last):
  File "/opt/conda/bin/ktransformers", line 8, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/main.py", line 134, in main
    create_interface(config=cfg, default_args=default_args)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/utils/create_interface.py", line 27, in create_interface
    GlobalInterface.interface = BackendInterface(default_args)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 20, in __init__
    self.tokenizer = AutoTokenizer.from_pretrained(args.model_dir,device = args.device)
  File "/opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 853, in from_pretrained
    config = AutoConfig.from_pretrained(
  File "/opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 975, in from_pretrained
    trust_remote_code = resolve_trust_remote_code(
  File "/opt/conda/lib/python3.10/site-packages/transformers/dynamic_module_utils.py", line 626, in resolve_trust_remote_code
    raise ValueError(
ValueError: The repository for /models/DeepSeek-V2-Coder contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co//models/DeepSeek-V2-Coder.
Please pass the argument `trust_remote_code=True` to allow custom code to be run.

UnicornChan commented 3 months ago

Perhaps you can download the GGUF file from this link: https://huggingface.co/LoneStriker/DeepSeek-Coder-V2-Instruct-GGUF/tree/main On the https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Instruct, there are only SafeTensor format files and the model's configs. about input parameter --gguf_path, there is only one GGUF format file placed in the directory.

#if your gguf path input as : --gguf_path /data/model/qwen2-57b-a14b-instruct-q4_k_m/
ls /data/model/qwen2-57b-a14b-instruct-q4_k_m/
#output
qwen2-57b-a14b-instruct-q4_k_m.gguf

about input parameter --model_path, it is where all the config files, excluding the SafeTensor model file, are stored.

#your model path input as : --model_path /data/model/Qwen2-57B-A14B-Instruct/
ls /data/model/Qwen2-57B-A14B-Instruct/
# output
config.json  generation_config.json  special_tokens_map.json  tokenizer_config.json  tokenizer.json  vocab.json

you can download all configs of model "DeepSeek-Coder-V2-Instruct" with GIT_LFS_SKIP_SMUDGE=1 git clone https://www.modelscope.cn/deepseek-ai/DeepSeek-Coder-V2-Instruct.git

UnicornChan commented 3 months ago

I recommend starting with a smaller model first: gguf file in：https://huggingface.co/mzwing/DeepSeek-V2-Lite-Chat-GGUF/tree/main model config file in: https://huggingface.co/deepseek-ai/DeepSeek-V2-Lite/tree/main (you can download by GIT_LFS_SKIP_SMUDGE=1 git clone https://www.modelscope.cn/deepseek-ai/DeepSeek-V2-Lite-Chat.git)

goldenquant commented 3 months ago

solved!thx.

kvcache-ai / ktransformers

using docker got errors #28