search

InternLM / lmdeploy

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.
https://lmdeploy.readthedocs.io/en/latest/
Apache License 2.0
4.59k stars 420 forks source link

[Bug] 我尝试部署InternLM-xcomposer-7b出现了模型架构的报错 #1421

Closed Zking668 closed 6 months ago

Zking668 commented 7 months ago

Checklist

Describe the bug

我按照教程运行example/vl/中的extract_xcomposer_llm.py和app.py。

报错为`KeyError: 'InternLMXComposerForCausalLM' Snipaste_2024-04-10_22-24-45

我想到了环境的问题，切换在Internstudio开发机切换cuda12.2镜像，依旧报错。 然后卸载掉lmdeploy重新安装pip install lmdeploy报错信息相同。

在issue中并未发现此问题，希望开发者团队回复。

Reproduction

python extract_xcomposer_llm.py python app.py --model-name internlm-xcomposer-7b --llm-ckpt internlm_model

Environment

sys.platform: linux
Python: 3.10.14 (main, Mar 21 2024, 16:24:04) [GCC 11.2.0]
CUDA available: True
MUSA available: False
numpy_random_seed: 2147483648
GPU 0: NVIDIA A100-SXM4-80GB
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.2, V12.2.140
GCC: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
PyTorch: 2.1.2
PyTorch compiling details: PyTorch built with:
  - GCC 9.3
  - C++ Version: 201703
  - Intel(R) oneAPI Math Kernel Library Version 2023.1-Product Build 20230303 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v3.1.1 (Git Hash 64f6bcbcbab628e96f33a62c3e975f8535a7bde4)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX512
  - CUDA Runtime 12.1
  - NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90
  - CuDNN 8.9.2
  - Magma 2.6.1
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=8.9.2, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-invalid-partial-specialization -Wno-unused-private-field -Wno-aligned-allocation-unavailable -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.1.2, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 

TorchVision: 0.16.2
LMDeploy: 0.3.0+b5b2b8b
transformers: 4.38.2
gradio: 3.50.2
fastapi: 0.110.1
pydantic: 2.6.4
triton: 2.1.0

Error traceback

Please following docs/install.md to install rotary_emb if you want to do fine-tuning
Init VIT ... Done
Init Perceive Sampler ... Done
Init InternLM ... Done
2024-04-10 22:05:16,312 - lmdeploy - WARNING - model_source: hf_model
Traceback (most recent call last):
  File "/root/code/lmdeploy/examples/vl/app.py", line 268, in <module>
    main()
  File "/root/code/lmdeploy/examples/vl/app.py", line 263, in main
    preprocessor, model = load_preprocessor_model(args)
  File "/root/code/lmdeploy/examples/vl/app.py", line 113, in load_preprocessor_model
    model = TurboMind.from_pretrained(llm_ckpt, model_name=args.model_name)
  File "/root/.conda/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/turbomind/turbomind.py", line 398, in from_pretrained
    return cls(model_path=pretrained_model_name_or_path,
  File "/root/.conda/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/turbomind/turbomind.py", line 167, in __init__
    self.model_comm = self._from_hf(model_source=model_source,
  File "/root/.conda/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/turbomind/turbomind.py", line 259, in _from_hf
    model_arch = get_model_from_config(model_path)
  File "/root/.conda/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/turbomind/utils.py", line 54, in get_model_from_config
    return ARCH_MAP[arch]
KeyError: 'InternLMXComposerForCausalLM
irexyc commented 7 months ago

这个是最初的vl的example，有点年久失修了，一些接口的变动没有及时同步到这里。你可以按照这个diff (https://github.com/irexyc/lmdeploy/commit/811f978c0e745f41969b91b6e94612884bff6e99) 来改一下安装目录下(/root/.conda/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/) 的文件。后面的话，vl examples 应该会用 vl pipeline 来实现。

Zking668 commented 7 months ago

感谢您认真的回复与解答，目前已经成功运行了！

我是大模型第二期实战营成员，正在学习LMDeploy部署，文档中只有使用LMDeploy运行视觉多模态大模型llava，并没有xcomposer，所以我就自己摸索过来了。那看来使用vl pipeline的demo还没有更新（手动狗头）。

目前还有一个问题，因为开发机上原本就有的xcomposer模型，所以我执行下面命令

python app.py --model-name internlm-xcomposer-7b --llm-ckpt /root/model/Shanghai_AI_Laboratory/internlm-xcomposer-7b

又产生了如下错误 aaaaaaa

我猜测是由于模型权重文件不一致导致的，所以我在extract_xcomposer_llm.py中发现在下载模型时会执行这部分代码 78979879879

所以我想问，使用lmdeploy部署模型必须要进行上述lora调整吗？

irexyc commented 7 months ago

这个报错是tokenizer加载报错了，不过我看hf上面internlm-xcompoer-7b的代码跟你这个log里的66行对不上。

image

https://huggingface.co/internlm/internlm-xcomposer-7b/blob/main/tokenization_InternLM_XComposer.py

按照这个log的话，下面这两句话可能也不会成功，感觉这个模型有问题。

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained('/root/model/Shanghai_AI_Laboratory/internlm-xcomposer-7b', trust_remote_code=True)

BTW, 你按照 vl examples 里面的命令是可以的吧？如果不改脚本的话，应该是会从hf上面下载最新的模型的。

Zking668 commented 7 months ago

我按照代码下载了最新的模型（4天前在autodl下载过）

import torch
from modelscope import snapshot_download, AutoModel, AutoTokenizer
import os
model_dir = snapshot_download('Shanghai_AI_Laboratory/internlm-xcomposer-7b', cache_dir='/root/autodl-tmp/model', revision='master')

ttttt

与昨天通过extract_xcomposer_llm.py下载的模型不一致 vvvvvvvvvvvvvvvv

irexyc commented 7 months ago

modelscope 上面的这个文件跟hf上的不一致，他的 self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs) 是在 super().__init__() 之后，可能当时的transformers 版本这样的逻辑是可以，现在不行了吧。你用从hf上面下载的就好了

https://www.modelscope.cn/models/Shanghai_AI_Laboratory/internlm-xcomposer-7b/file/view/master/tokenization_InternLM_XComposer.py?status=1

Zking668 commented 7 months ago

我尝试使用 serve api_server 部署xcomposer

lmdeploy serve api_server \
    /root/code/lmdeploy/examples/vl/internlm_model \
    --model-format hf \
    --quant-policy 0 \
    --server-name 0.0.0.0 \
    --server-port 6006 \
    --tp 1

运行上述命令，可以打开Fastapi接口网页 Snipaste_2024-04-11_20-22-23 然后我接着先GET获取模型名称base，再执行POST chat/completion产生错误 image image

请问现在server api还不能部署xcomposer吗？

irexyc commented 7 months ago

@Zking668

嗯，上面有提到，目前 vl pipeline (vl server) 不支持 xcomposer。

目前支持部署的有 qwen-vl, llava, yi-vl, deepseekvl, internvl(pr中) 这些。

Zking668 commented 7 months ago

好的好的，感谢您一直耐心回复！

另外我想请教一下，如果仅仅是把xcomposer实现FastApi的部署，类似于VisualGLM-6B中的api_hf.py ，是可以实现的吗，需要做一些什么修改呢？

irexyc commented 6 months ago

如果你只是想要一个简单的web api，可以按照 VisualGLM-6B 的改吧。

推理用transformers，用法可以参照xcomposer的repo，

fastapi传参你可以参考官方的tutorial https://fastapi.tiangolo.com/tutorial/body/

复杂一点的，比如兼容openai格式的，可以参考 https://github.com/QwenLM/Qwen/blob/main/openai_api.py

201lichunqiang commented 6 months ago

我这边在部署internlm-xcomposer-7b的时候，遇到了一些问题。显示AttributeError: 'InternLMXComposerTokenizer' object has no attribute 'chat_template'

(vqa) root@I198d13b00000901dff:~/model/Shanghai_AI_Laboratory/internlm-xcomposer-7b# lmdeploy serve api_server \

/root/model/Shanghai_AI_Laboratory/internlm-xcomposer-7b \ --model-format hf \ --quant-policy 0 \ --server-name 0.0.0.0 \ --server-port 23333 \ --tp 1 Traceback (most recent call last): File "/usr/local/miniconda3/lib/python3.8/site-packages/lmdeploy/model.py", line 1318, in best_match_model if tokenizer_config.chat_template is None: AttributeError: 'InternLMXComposerTokenizer' object has no attribute 'chat_template'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/miniconda3/bin/lmdeploy", line 8, in sys.exit(run()) File "/usr/local/miniconda3/lib/python3.8/site-packages/lmdeploy/cli/entrypoint.py", line 37, in run args.run(args) File "/usr/local/miniconda3/lib/python3.8/site-packages/lmdeploy/cli/serve.py", line 283, in api_server run_api_server(args.model_path, File "/usr/local/miniconda3/lib/python3.8/site-packages/lmdeploy/serve/openai/api_server.py", line 1217, in serve VariableInterface.async_engine = pipeline_class( File "/usr/local/miniconda3/lib/python3.8/site-packages/lmdeploy/serve/async_engine.py", line 182, in init self.model_name = deduce_a_name(model_path, model_name, backend_config, File "/usr/local/miniconda3/lib/python3.8/site-packages/lmdeploy/serve/async_engine.py", line 58, in deduce_a_name model_name = best_match_model(model_path) File "/usr/local/miniconda3/lib/python3.8/site-packages/lmdeploy/model.py", line 1321, in best_match_model assert type(e) == OSError AssertionError

irexyc commented 6 months ago

@201lichunqiang

不支持 internlm-xcompoer的部署，支持internlm-xcomposer2的部署 （lmdeploy >= 0.4.0)

201lichunqiang commented 6 months ago

懂了，这就去试试xcomposer2

201lichunqiang commented 6 months ago

internlm-xcomposer2是只支持创作图文并茂文章吗 image

irexyc commented 6 months ago

@201lichunqiang

我理解 internlm-xcomposer2-vl 跟其他的多模态模型一样，internlm-xcomposer2 是在 internlm-xcomposer2-vl 的基础上强化了 图文创作的功能。具体的你可以去他们的 repo 问一下。

201lichunqiang commented 6 months ago

在本地服务器运行了几次internlm-xcomposer2-vl-7b，还是报错，python /home/tjut_chunqiangli/root/demo/InternLM-XComposer/examples/gradio_demo_composition.py \

--code_path /home/tjut_chunqiangli/root/model/internlm-xcomposer2-vl-7b \ --private \ --num_gpus 8 \ --port 6006 You are using a model of type internlmxcomposer2 to instantiate a model of type internlm. This is not supported for all configurations of models and can yield errors. /home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/transformers/utils/generic.py:311: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead. torch.utils._pytree._register_pytree_node( /home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/transformers/utils/generic.py:311: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead. torch.utils._pytree._register_pytree_node( Set max length to 4096 Traceback (most recent call last): File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/urllib3/connection.py", line 198, in _new_conn sock = connection.create_connection( File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection raise err File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/urllib3/util/connection.py", line 73, in create_connection sock.connect(sa) OSError: [Errno 101] Network is unreachable

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/urllib3/connectionpool.py", line 793, in urlopen response = self._make_request( File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/urllib3/connectionpool.py", line 491, in _make_request raise new_e File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/urllib3/connectionpool.py", line 467, in _make_request self._validate_conn(conn) File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn conn.connect() File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/urllib3/connection.py", line 616, in connect self.sock = sock = self._new_conn() File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/urllib3/connection.py", line 213, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7fd973cef280>: Failed to establish a new connection: [Errno 101] Network is unreachable

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/urllib3/connectionpool.py", line 847, in urlopen retries = retries.increment( File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/urllib3/util/retry.py", line 515, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /openai/clip-vit-large-patch14-336/resolve/main/config.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fd973cef280>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1261, in hf_hub_download metadata = get_hf_file_metadata( File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 119, in _inner_fn return fn(args, kwargs) File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1674, in get_hf_file_metadata r = _request_wrapper( File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 369, in _request_wrapper response = _request_wrapper( File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 392, in _request_wrapper response = get_session().request(method=method, url=url, params) File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, send_kwargs) File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, kwargs) File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 68, in send return super().send(request, args, **kwargs) File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/requests/adapters.py", line 519, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /openai/clip-vit-large-patch14-336/resolve/main/config.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fd973cef280>: Failed to establish a new connection: [Errno 101] Network is unreachable'))"), '(Request ID: 067247cb-5ce1-4cd9-878d-6e8079a62bc7)')

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/transformers/utils/hub.py", line 429, in cached_file resolved_file = hf_hub_download( File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 119, in _inner_fn return fn(*args, **kwargs) File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1406, in hf_hub_download raise LocalEntryNotFoundError( huggingface_hub.utils._errors.LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/tjut_chunqiangli/root/demo/InternLM-XComposer/examples/gradio_demo_composition.py", line 842, in demo_ui = Demo_UI(args.code_path, args.num_gpus) File "/home/tjut_chunqiangli/root/demo/InternLM-XComposer/examples/gradio_demo_composition.py", line 155, in init self.model = AutoModelForCausalLM.from_pretrained(code_path, device_map='cuda', trust_remote_code=True).half().eval() File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 560, in from_pretrained return model_class.from_pretrained( File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3085, in from_pretrained model = cls(config, *model_args, model_kwargs) File "/home/tjut_chunqiangli/.cache/huggingface/modules/transformers_modules/internlm-xcomposer2-vl-7b/modeling_internlm_xcomposer2.py", line 67, in init self.vit = build_vision_tower() File "/home/tjut_chunqiangli/.cache/huggingface/modules/transformers_modules/internlm-xcomposer2-vl-7b/build_mlp.py", line 11, in build_vision_tower return CLIPVisionTower(vision_tower) File "/home/tjut_chunqiangli/.cache/huggingface/modules/transformers_modules/internlm-xcomposer2-vl-7b/build_mlp.py", line 58, in init self.load_model() File "/home/tjut_chunqiangli/.cache/huggingface/modules/transformers_modules/internlm-xcomposer2-vl-7b/build_mlp.py", line 62, in load_model self.vision_tower = CLIPVisionModel.from_pretrained( File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2662, in from_pretrained config, model_kwargs = cls.config_class.from_pretrained( File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/transformers/models/clip/configuration_clip.py", line 238, in from_pretrained config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, kwargs) File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/transformers/configuration_utils.py", line 620, in get_config_dict config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs) File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/transformers/configuration_utils.py", line 675, in _get_config_dict resolved_config_file = cached_file( File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/transformers/utils/hub.py", line 469, in cached_file raise EnvironmentError( OSError: We couldn't connect to 'https://huggingface.co' to load this file, couldn't find it in the cached files and it looks like openai/clip-vit-large-patch14-336 is not the path to a directory containing a file named config.json. Checkout your internet connection or see how to run the library in offline mode at 'https://huggingface.co/docs/transformers/installation#offline-mode'.

irexyc commented 6 months ago

@201lichunqiang

在本地服务器运行了几次internlm-xcomposer2-vl-7b，还是报错，python /home/tjut_chunqiangli/root/demo/InternLM-XComposer/examples/gradio_demo_composition.py \

这是internlm-xcomposer2 他们 repo 提供的gradio demo，去他们的 repo下面问比较好。

我们提供的demo 方式是 lmdeploy serve gradio internlm/internlm-xcomposer2-vl-7b

201lichunqiang commented 6 months ago

我这边模型是从huggingface上下载下来，传输到服务器的，然后用命令运行， lmdeploy serve api_server\
/home/tjut_chunqiangli/root/model/internlm-xcomposer2-vl-7b \ --model-format hf \ --quant-policy 0 \ --server-name 0.0.0.0 \ --server-port 23333 \ --tp 4 还是出现这个问题，(xiangmu) tjut_chunqiangli@taizun-SYS-4029GP-TRT:~/root/model/internlm-xcomposer2-vl-7b$ lmdeploy serve api_server /home/tjut_chunqiangli/root/model/internlm-xcomposer2-vl-7b --model-format hf --quant-policy 0 --server-name 0.0.0.0 --server-port 23333 --tp 4 You are using a model of type internlmxcomposer2 to instantiate a model of type internlm. This is not supported for all configurations of models and can yield errors. You are using a model of type internlmxcomposer2 to instantiate a model of type internlm. This is not supported for all configurations of models and can yield errors. /home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/transformers/utils/generic.py:311: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead. torch.utils._pytree._register_pytree_node( You are using a model of type internlmxcomposer2 to instantiate a model of type internlm. This is not supported for all configurations of models and can yield errors. Set max length to 4096 Traceback (most recent call last): File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/urllib3/connection.py", line 198, in _new_conn sock = connection.create_connection( File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection raise err File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/urllib3/util/connection.py", line 73, in create_connection sock.connect(sa) OSError: [Errno 101] Network is unreachable

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/urllib3/connectionpool.py", line 793, in urlopen response = self._make_request( File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/urllib3/connectionpool.py", line 491, in _make_request raise new_e File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/urllib3/connectionpool.py", line 467, in _make_request self._validate_conn(conn) File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn conn.connect() File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/urllib3/connection.py", line 616, in connect self.sock = sock = self._new_conn() File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/urllib3/connection.py", line 213, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f460f97c1c0>: Failed to establish a new connection: [Errno 101] Network is unreachable

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/urllib3/connectionpool.py", line 847, in urlopen retries = retries.increment( File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/urllib3/util/retry.py", line 515, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /openai/clip-vit-large-patch14-336/resolve/main/config.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f460f97c1c0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1232, in hf_hub_download metadata = get_hf_file_metadata( File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn return fn(args, kwargs) File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1599, in get_hf_file_metadata r = _request_wrapper( File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 417, in _request_wrapper response = _request_wrapper( File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 452, in _request_wrapper return http_backoff( File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 258, in http_backoff response = session.request(method=method, url=url, kwargs) File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, send_kwargs) File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, kwargs) File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 63, in send return super().send(request, args, **kwargs) File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/requests/adapters.py", line 519, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /openai/clip-vit-large-patch14-336/resolve/main/config.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f460f97c1c0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))"), '(Request ID: fbc501e5-ef5a-46f4-bcf4-336b9316f267)')

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/transformers/utils/hub.py", line 429, in cached_file resolved_file = hf_hub_download( File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn return fn(*args, **kwargs) File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1349, in hf_hub_download raise LocalEntryNotFoundError( huggingface_hub.utils._errors.LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/bin/lmdeploy", line 8, in sys.exit(run()) File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/lmdeploy/cli/entrypoint.py", line 37, in run args.run(args) File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/lmdeploy/cli/serve.py", line 283, in api_server run_api_server(args.model_path, File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/lmdeploy/serve/openai/api_server.py", line 1217, in serve VariableInterface.async_engine = pipeline_class( File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/lmdeploy/serve/vl_async_engine.py", line 16, in init self.vl_encoder = ImageEncoder(model_path) File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/lmdeploy/vl/engine.py", line 68, in init self.model = load_vl_model(model_path) File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/lmdeploy/vl/model/builder.py", line 40, in load_vl_model return Xcomposer2VisionModel(model_path) File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/lmdeploy/vl/model/xcomposer2.py", line 20, in init self.build_model() File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/lmdeploy/vl/model/xcomposer2.py", line 28, in build_model model = AutoModelForCausalLM.from_config(config, File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 438, in from_config return model_class._from_config(config, kwargs) File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/transformers/modeling_utils.py", line 1190, in _from_config model = cls(config, kwargs) File "/home/tjut_chunqiangli/.cache/huggingface/modules/transformers_modules/internlm-xcomposer2-vl-7b/modeling_internlm_xcomposer2.py", line 67, in init self.vit = build_vision_tower() File "/home/tjut_chunqiangli/.cache/huggingface/modules/transformers_modules/internlm-xcomposer2-vl-7b/build_mlp.py", line 11, in build_vision_tower return CLIPVisionTower(vision_tower) File "/home/tjut_chunqiangli/.cache/huggingface/modules/transformers_modules/internlm-xcomposer2-vl-7b/build_mlp.py", line 58, in init self.load_model() File "/home/tjut_chunqiangli/.cache/huggingface/modules/transformers_modules/internlm-xcomposer2-vl-7b/build_mlp.py", line 62, in load_model self.vision_tower = CLIPVisionModel.from_pretrained( File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2662, in from_pretrained config, model_kwargs = cls.config_class.from_pretrained( File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/transformers/models/clip/configuration_clip.py", line 238, in from_pretrained config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, kwargs) File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/transformers/configuration_utils.py", line 620, in get_config_dict config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, kwargs) File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/transformers/configuration_utils.py", line 675, in _get_config_dict resolved_config_file = cached_file( File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/transformers/utils/hub.py", line 469, in cached_file raise EnvironmentError( OSError: We couldn't connect to 'https://huggingface.co' to load this file, couldn't find it in the cached files and it looks like openai/clip-vit-large-patch14-336 is not the path to a directory containing a file named config.json. Checkout your internet connection or see how to run the library in offline mode at 'https://huggingface.co/docs/transformers/installation#offline-mode'.

irexyc commented 6 months ago

@201lichunqiang

从log看，是因为你的服务器没有网络，或者无法访问 huggingface 导致这行代码报错：

model = AutoModelForCausalLM.from_config(config, trust_remote_code=True)

这应该是transformers的问题，你可以试下用 transformers 的代码跑应该也会报这个错： https://huggingface.co/internlm/internlm-xcomposer2-7b

解决方式就是添加环境变量，transformers 提供 offline-mode https://huggingface.co/docs/transformers/installation#offline-mode

export TRANSFORMERS_OFFLINE=1
lmdeploy serve api_server ...
201lichunqiang commented 6 months ago

我这边尝试从训练营2期的第一个demo的视频中，模型中 .py 后缀的代码，替换掉，我在hugging face中下载的代码，模型加载有一部分进度。 You are using a model of type internlmxcomposer2 to instantiate a model of type internlm. This is not supported for all configurations of models and can yield errors. Set max length to 4096 2024-04-25 14:51:58,903 - modelscope - INFO - PyTorch version 2.2.2 Found. 2024-04-25 14:51:58,905 - modelscope - INFO - Loading ast index from /home/tjut_chunqiangli/.cache/modelscope/ast_indexer 2024-04-25 14:51:58,945 - modelscope - INFO - No valid ast index found from /home/tjut_chunqiangli/.cache/modelscope/ast_indexer, generating ast index from prebuilt! 2024-04-25 14:51:59,001 - modelscope - INFO - Loading done! Current index file version is 1.9.5, with md5 db856e71c754e2615b3de786ae5b33b1 and a total number of 945 components indexed Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████| 4.65k/4.65k [00:00<00:00, 15.3MB/s] Downloading: 100%|███████████████████████████████████████████████████████████████████████████████████████| 49.0/49.0 [00:00<00:00, 335kB/s] Downloading: 100%|██████████████████████████████████████████████████████████████████████████████████████| 512k/512k [00:00<00:00, 9.80MB/s] Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████| 316/316 [00:00<00:00, 1.82MB/s] Downloading: 100%|███████████████████████████████████████████████████████████████████████████████████▉| 1.59G/1.59G [00:17<00:00, 96.0MB/s] Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████| 1.02k/1.02k [00:00<00:00, 3.50MB/s] Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████| 389/389 [00:00<00:00, 2.26MB/s] Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████▉| 1.59G/1.59G [00:16<00:00, 106MB/s] Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████| 2.12M/2.12M [00:00<00:00, 20.5MB/s] Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████| 844/844 [00:00<00:00, 4.32MB/s] Downloading: 100%|██████████████████████████████████████████████████████████████████████████████████████| 842k/842k [00:00<00:00, 13.2MB/s] Position interpolate from 24x24 to 35x35 Position interpolate from 24x24 to 35x35 Traceback (most recent call last): File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/bin/lmdeploy", line 8, in sys.exit(run()) File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/lmdeploy/cli/entrypoint.py", line 37, in run args.run(args) File "/home/tjut_chunqiangli/anaconda3/envs/xiangmu/lib/python3.10/site-packages/lmdeploy/cli/serve.py", line 283, in api_server

201lichunqiang commented 6 months ago

我这边torch只能到2.0.1，lmdeploy 应该用哪个版本，可以部署，internlm-xcomposer2-vl-7b

irexyc commented 6 months ago

目前多模态模型，LLM 的后端用的是 turbomind 不是 pytorch，vision的部分才会用到 pytorch。

vision的部分对pytorch版本无要求。

201lichunqiang commented 6 months ago

我这边pip install lmdeploy==0.4.0 之后torch就变成了2.2.2，我这边cuda是11.7，对应torch2.0.1

irexyc commented 6 months ago

pytorch 和 lmdeploy 依赖的 cuda 大版本需要一致，不然可能会有问题。

pip install lmdeploy 从0.3.0 开始装的都是 cuda 12编译的版本。

你如果要装 cuda 11 编译的版本，可以这样按照下面的方式，README里面有写：https://github.com/InternLM/lmdeploy?tab=readme-ov-file#installation

export LMDEPLOY_VERSION=0.4.0
export PYTHON_VERSION=38
pip install https://github.com/InternLM/lmdeploy/releases/download/v${LMDEPLOY_VERSION}/lmdeploy-${LMDEPLOY_VERSION}+cu118-cp${PYTHON_VERSION}-cp${PYTHON_VERSION}-manylinux2014_x86_64.whl --extra-index-url https://download.pytorch.org/whl/cu118
201lichunqiang commented 6 months ago

我自己又找了个服务器跑了一下，成功了，感谢！ ![Uploading image.png…]()

201lichunqiang commented 6 months ago

我最近研究部署的其他模型的时候，遇到了这个问题，Segmentation fault (core dumped)

201lichunqiang commented 5 months ago

对于本地部署上海人工智能实验室的Internlm=xcomposer2-vl-7b模型，总结一下流程。 服务器选择cuda12.1 之后执行一下的命令 pip install huggingface-hub==0.17.3 pip install transformers==4.34 pip install psutil==5.9.8 pip install accelerate==0.24.1 pip install streamlit==1.32.2 pip install matplotlib==3.8.3 pip install modelscope==1.9.5 pip install sentencepiece==0.1.99

pip install timm==0.4.12 # 此版本应该兼容 pip install sentencepiece==0.1.99 # 此版本应该兼容 pip install markdown2==2.4.10 # 此版本应该兼容 pip install xlsxwriter==3.1.2 # 此版本应该兼容 pip install gradio==4.13.0 # 此版本应该兼容 pip install modelscope==1.9.5 # 需要确认兼容性，暂无具体版本依赖信息

如果使用imdeploy0.3.0，torch版本会变成2.12，这时需要再改变一下torchversion的版本改成 0.16.2+cu121 pip install lmdeploy[all]==0.3.0 或者 pip install lmdeploy[all]==0.4.0

模型这边，我最初是使用的Hugging Face中的，但是出现了一些问题，之后我的做法是，把Hugging Face中下载的模型，把其中以.py结尾的文件全部替换成了，浦语提供的开发机中，Share内对应的.py文件。

最后一步使用Imdeploy部署模型 lmdeploy serve api_server \ /root/model/internlm-xcomposer2-vl-7b \ --model-format hf \ --quant-policy 0 \ --server-name 0.0.0.0 \ --server-port 23333 \ --tp 2