[Bug] MiniCPMV的推理有问题

[LDLINGLINGLING]

Checklist

• [X] 1. I have searched related issues but cannot get the expected help.
• [X] 2. The bug has not been fixed in the latest version.

Describe the bug

官方的给的python示例代码推理Minicpmv-2.5报错

Reproduction

from lmdeploy import pipeline from lmdeploy.vl import load_image

pipe = pipeline('/root/ld/ld_model_pretrain/MiniCPM-Llama3-V-2_5')

image = load_image('https://wxls-cms.oss-cn-hangzhou.aliyuncs.com/online/2024-04-18/218da022-f4bf-456a-99af-5cb8e157f7b8.jpg') response = pipe(('describe this image', image)) print(response)

Environment

absl-py                       2.1.0
accelerate                    0.30.1
addict                        2.4.0
aiobotocore                   2.7.0
aiofiles                      23.2.1
aiohttp                       3.9.5
aioitertools                  0.11.0
aioprometheus                 23.12.0
aiosignal                     1.3.1
aliyun-python-sdk-core        2.15.1
aliyun-python-sdk-kms         2.16.3
altair                        5.3.0
annotated-types               0.7.0
anyio                         4.3.0
asttokens                     2.4.1
async-timeout                 4.0.3
attrs                         23.2.0
auto_gptq                     0.8.0.dev0+cu121
autoawq                       0.2.5+cu121
autoawq_kernels               0.0.6
bcrypt                        4.1.3
bitsandbytes                  0.43.1
blinker                       1.8.2
boto3                         1.34.129
botocore                      1.31.64
Brotli                        1.0.9
cachetools                    5.3.3
certifi                       2024.6.2
cffi                          1.16.0
charset-normalizer            2.0.4
click                         8.1.7
cloudpickle                   3.0.0
cmake                         3.29.3
cn2an                         0.5.22
colorama                      0.4.6
comm                          0.2.2
contourpy                     1.2.1
cpm-kernels                   1.0.11
crcmod                        1.7
cryptography                  42.0.7
cycler                        0.12.1
dataclasses-json              0.6.6
datasets                      2.18.0
debugpy                       1.6.7
decorator                     5.1.1
deepspeed                     0.14.4
dill                          0.3.8
distro                        1.9.0
dnspython                     2.6.1
docstring_parser              0.16
e                             1.4.5
ecdsa                         0.19.0
einops                        0.5.0
email_validator               2.1.1
entrypoints                   0.4
et-xmlfile                    1.1.0
evaluate                      0.4.2
exceptiongroup                1.2.0
executing                     2.0.1
fairscale                     0.4.13
fastapi                       0.110.3
fastapi-cli                   0.0.4
ffmpy                         0.3.2
filelock                      3.13.1
fire                          0.6.0
fonttools                     4.51.0
frozenlist                    1.4.1
fsspec                        2023.10.0
func_timeout                  4.3.5
fuzzywuzzy                    0.18.0
gast                          0.5.4
gekko                         1.1.1
gitdb                         4.0.11
GitPython                     3.1.43
gmpy2                         2.1.2
gradio                        4.26.0
gradio_client                 0.15.1
greenlet                      3.0.3
h11                           0.14.0
hjson                         3.1.0
httpcore                      1.0.5
httptools                     0.6.1
httpx                         0.27.0
huggingface-hub               0.23.1
idna                          3.7
immutabledict                 4.2.0
importlib_metadata            7.1.0
importlib_resources           6.4.0
ipykernel                     6.29.4
ipython                       8.26.0
jedi                          0.19.1
jieba                         0.42.1
Jinja2                        3.1.3
jmespath                      0.10.0
joblib                        1.4.2
json5                         0.9.25
jsonpatch                     1.33
jsonpointer                   2.4
jsonschema                    4.22.0
jsonschema-specifications     2023.12.1
jupyter-client                7.3.4
jupyter_core                  5.7.2
kiwisolver                    1.4.5
langchain                     0.2.1
langchain-community           0.2.1
langchain-core                0.2.1
langchain-text-splitters      0.2.0
langdetect                    1.0.9
langsmith                     0.1.63
Levenshtein                   0.25.1
lmdeploy                      0.5.0
loguru                        0.7.2
ltp                           4.2.14
ltp-core                      0.1.4
ltp-extension                 0.1.13
lxml                          5.2.2
markdown-it-py                3.0.0
MarkupSafe                    2.1.3
marshmallow                   3.21.2
matplotlib                    3.9.0
matplotlib-inline             0.1.7
mdurl                         0.1.2
mkl-fft                       1.3.8
mkl-random                    1.2.4
mkl-service                   2.4.0
mlx                           0.2.0
mmengine-lite                 0.10.4
modelscope                    1.14.0
mpmath                        1.3.0
multidict                     6.0.5
multiprocess                  0.70.16
mypy-extensions               1.0.0
nest_asyncio                  1.6.0
networkx                      3.1
ninja                         1.11.1.1
nltk                          3.8
numpy                         1.26.4
nvidia-ml-py                  12.555.43
openai                        1.34.0
OpenCC                        1.1.7
opencompass                   0.2.5
opencv-contrib-python         4.10.0.84
opencv-python-headless        4.10.0.84
openpyxl                      3.1.4
orjson                        3.10.3
oss2                          2.18.5
packaging                     23.2
pandas                        1.5.3
parso                         0.8.4
passlib                       1.7.4
peft                          0.11.1
pexpect                       4.9.0
pickleshare                   0.7.5
pillow                        10.3.0
pip                           24.0
platformdirs                  4.2.2
portalocker                   2.8.2
prettytable                   3.10.0
proces                        0.1.7
prompt_toolkit                3.0.47
protobuf                      5.27.1
psutil                        5.9.0
ptyprocess                    0.7.0
pure-eval                     0.2.2
py-cpuinfo                    9.0.0
pyarrow                       16.1.0
pyarrow-hotfix                0.6
pyasn1                        0.6.0
pycparser                     2.22
pycryptodome                  3.20.0
pydantic                      2.7.1
pydantic_core                 2.18.2
pydeck                        0.9.1
pydub                         0.25.1
pyext                         0.7
Pygments                      2.18.0
pynvml                        11.5.0
pyparsing                     3.1.2
pypinyin                      0.51.0
PySocks                       1.7.1
python-dateutil               2.9.0
python-dotenv                 1.0.1
python-jose                   3.3.0
python-Levenshtein            0.25.1
python-multipart              0.0.9
pytz                          2024.1
PyYAML                        6.0.1
pyzmq                         25.1.2
quantile-python               1.1
rank-bm25                     0.2.2
rapidfuzz                     3.9.3
referencing                   0.35.1
regex                         2024.5.15
requests                      2.31.0
rich                          13.7.1
rouge                         1.0.1
rouge-chinese                 1.0.3
rouge_score                   0.1.2
rpds-py                       0.18.1
rsa                           4.9
ruff                          0.4.5
s3fs                          2023.10.0
s3transfer                    0.10.1
sacrebleu                     2.4.2
safetensors                   0.4.3
scikit-learn                  1.2.1
scipy                         1.13.1
seaborn                       0.13.2
semantic-version              2.10.0
sentence-transformers         3.0.1
sentencepiece                 0.2.0
setuptools                    69.5.1
shellingham                   1.5.4
shortuuid                     1.0.13
shtab                         1.7.1
simplejson                    3.19.2
six                           1.16.0
smmap                         5.0.1
sniffio                       1.3.1
sortedcontainers              2.4.0
SQLAlchemy                    2.0.30
sse-starlette                 2.1.2
stack-data                    0.6.2
starlette                     0.37.2
streamlit                     1.36.0
sympy                         1.12
tabulate                      0.9.0
tblib                         3.0.0
tenacity                      8.3.0
tensorboardX                  2.6.2.2
termcolor                     2.4.0
threadpoolctl                 3.5.0
tiktoken                      0.7.0
timeout-decorator             0.5.0
timm                          1.0.7
tokenizers                    0.19.1
toml                          0.10.2
tomli                         2.0.1
tomlkit                       0.12.0
toolz                         0.12.1
torch                         2.1.1
torchaudio                    2.1.1
torchvision                   0.16.1
tornado                       6.1
tqdm                          4.64.1
traitlets                     5.14.3
transformers                  4.41.2
transformers-stream-generator 0.0.5
triton                        2.1.0
trl                           0.8.6
typer                         0.11.1
typing_extensions             4.11.0
typing-inspect                0.9.0
tyro                          0.8.4
tzdata                        2024.1
ujson                         5.10.0
urllib3                       2.0.7
uvicorn                       0.29.0
uvloop                        0.19.0
vllm-flash-attn               2.5.8.post2
watchdog                      4.0.1
watchfiles                    0.21.0
wcwidth                       0.2.13
websockets                    11.0.3
wheel                         0.43.0
wrapt                         1.16.0
xinference                    0.12.1+10.g5cef7c3
xoscar                        0.3.0
xxhash                        3.4.1
yapf                          0.40.2
yarl                          1.9.4
zipp                          3.19.0
zstandard                     0.22.0

Error traceback

错误信息如下：
2024-07-10 15:33:19,463 - lmdeploy - ERROR - Engine loop failed with error: MiniCPMV.forward() missing 1 required positional argument: 'data'
Traceback (most recent call last):
  File "/root/ld/ld_project/lmdeploy/lmdeploy/pytorch/engine/request.py", line 17, in _raise_exception_on_finish
    task.result()
  File "/root/ld/ld_project/lmdeploy/lmdeploy/pytorch/engine/engine.py", line 857, in async_loop
    await self._async_loop()
  File "/root/ld/ld_project/lmdeploy/lmdeploy/pytorch/engine/engine.py", line 847, in _async_loop
    await __step(True)
  File "/root/ld/ld_project/lmdeploy/lmdeploy/pytorch/engine/engine.py", line 833, in __step
    raise e
  File "/root/ld/ld_project/lmdeploy/lmdeploy/pytorch/engine/engine.py", line 825, in __step
    raise out
  File "/root/ld/ld_project/lmdeploy/lmdeploy/pytorch/engine/engine.py", line 774, in _async_loop_background
    await self._async_step_background(
  File "/root/ld/ld_project/lmdeploy/lmdeploy/pytorch/engine/engine.py", line 683, in _async_step_background
    output = await self._async_model_forward(inputs,
  File "/root/ld/ld_project/lmdeploy/lmdeploy/utils.py", line 253, in __tmp
    return (await func(*args, **kwargs))
  File "/root/ld/ld_project/lmdeploy/lmdeploy/pytorch/engine/engine.py", line 590, in _async_model_forward
    return await __forward(inputs)
  File "/root/ld/ld_project/lmdeploy/lmdeploy/pytorch/engine/engine.py", line 568, in __forward
    return await self.model_agent.async_forward(
  File "/root/ld/ld_project/lmdeploy/lmdeploy/pytorch/engine/model_agent.py", line 756, in async_forward
    output = self._forward_impl(inputs,
  File "/root/ld/ld_project/lmdeploy/lmdeploy/pytorch/engine/model_agent.py", line 723, in _forward_impl
    output = model_forward(
  File "/root/ld/ld_project/lmdeploy/lmdeploy/pytorch/engine/model_agent.py", line 497, in model_forward
    output = patched_model.patched_forward(
  File "/root/ld/ld_project/lmdeploy/lmdeploy/pytorch/models/patch.py", line 210, in __call__
    output = self._model(*args, **kwargs)
  File "/root/ocnda/envs/minicpm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/root/ocnda/envs/minicpm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
TypeError: MiniCPMV.forward() missing 1 required positional argument: 'data'

[irexyc]

怎么安装的 lmdeploy ？pip install . 是不行的。

正常不会用pytorch backend来加载模型。

[LDLINGLINGLING]

那应该如何安装

[irexyc]

https://github.com/InternLM/lmdeploy/blob/main/docs/zh_cn/multi_modal/minicpmv.md

新建一个环境，然后 pip install lmdeploy

你可以按照这个说明来

[LDLINGLINGLING]

(minicpmv) (base) root@gpu-62:~/ld# /root/ocnda/envs/minicpmv/bin/python /root/ld/ld_project/MiniCPM-V/lmdeploy_cli.py Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. [WARNING] gemm_config.in is not found; using default GEMM algo
卡在这里不动了，是不能使用本地pytorch路径进行加载？

[irexyc]

创建的这一行改成这个把日志打开。 pipe = pipeline('/root/ld/ld_model_pretrain/MiniCPM-Llama3-V-2_5', log_level='INFO')

然后把完整的日志发一下。

[irexyc]

https://github.com/InternLM/lmdeploy/issues/2004#issuecomment-2224402209

可以按照这个说明改一下这一行，run_in_executor的时候，没有加异常处理，可能会隐藏真实的问题。

[Wimen]

同遇到这个问题，日志如下

[irexyc]

@Wimen

minicpmv 在 turbomind 引擎支持的。

turbomind 引擎需要编译，只靠 pip install . 是无法安装的。

可以在这里安装最新的版本 https://github.com/zhyncs/lmdeploy-build/releases

[Wimen]

嗯，之前看到了前面的评论，所以我是新起的环境，直接pip install lmdeploy装了0.5.0，另外里面有报transformers的版本warnings，但还是会出现这个问题

[Wimen]

请问有什么办法可以预先确认tubormind是可以正常工作的

[irexyc]

请问有什么办法可以预先确认tubormind是可以正常工作的

from lmdeploy.turbomind import *

不报错的话，是成功的，报错的话，可以检查下原因：

python -c "import lmdeploy; print(lmdeploy)"
<module 'lmdeploy' from '/home/chenxin/ws3/vl/lmdeploy/__init__.py'>

ldd /home/chenxin/ws3/vl/lmdeploy/lib/*

linux-vdso.so.1 (0x00007ffe6c7ad000)
libnccl.so.2 => /usr/lib/x86_64-linux-gnu/libnccl.so.2 (0x00007ff8c557f000)
libcudart.so.11.0 => /usr/local/cuda/lib64/libcudart.so.11.0 (0x00007ff8c52d8000)
libcublas.so.11 => /usr/local/cuda/lib64/libcublas.so.11 (0x00007ff8bf67a000)
libcublasLt.so.11 => /usr/local/cuda/lib64/libcublasLt.so.11 (0x00007ff89b0f4000)
libdl.so.2 => /usr/lib/x86_64-linux-gnu/libdl.so.2 (0x00007ff89b0ec000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007ff89af0a000)
libm.so.6 => /usr/lib/x86_64-linux-gnu/libm.so.6 (0x00007ff89adbb000)
libgcc_s.so.1 => /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ff89ada0000)
libc.so.6 => /usr/lib/x86_64-linux-gnu/libc.so.6 (0x00007ff89abae000)
/lib64/ld-linux-x86-64.so.2 (0x00007ff8dee41000)
libpthread.so.0 => /usr/lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff89ab8b000)
librt.so.1 => /usr/lib/x86_64-linux-gnu/librt.so.1 (0x00007ff89ab7f000)

pip instal lmdeploy 默认会装cuda 12的 runtime，可以用 nvidia-smi 检查下驱动支持的cuda版本

[LDLINGLINGLING]

日志如下： 2024-07-12 17:21:55,573 - lmdeploy - INFO - start ImageEncoder._forward_loop 2024-07-12 17:21:55,573 - lmdeploy - INFO - ImageEncoder received 1 images, left 1 images. 2024-07-12 17:21:55,573 - lmdeploy - INFO - ImageEncoder process 1 images, left 0 images. 在这里卡住了，而且问题狠奇怪，tp>1能跑，tp=1卡在以下bug [WARNING] gemm_config.in is not found; using default GEMM algo

[irexyc]

@LDLINGLINGLING

可以把不能跑的图传上来看看

另外，可以按照这个 comment改一下代码看看 https://github.com/InternLM/lmdeploy/issues/2004#issuecomment-2224402209

[github-actions[bot]]

This issue is marked as stale because it has been marked as invalid or awaiting response for 7 days without any further response. It will be closed in 5 days if the stale label is not removed or if there is no further response.

[github-actions[bot]]

This issue is closed because it has been stale for 5 days. Please open a new issue if you have similar issues or you have any new updates now.

[Howe-Young]

https://github.com/InternLM/lmdeploy/blob/main/docs/zh_cn/multi_modal/minicpmv.md

新建一个环境，然后 pip install lmdeploy

你可以按照这个说明来

支持openbmb/MiniCPM-V-2这个模型吗？我直接推理会报错：

AttributeError: 'MiniCPMVConfig' object has no attribute 'slice_mode'

[lvhan028]

支持的模型都在 https://lmdeploy.readthedocs.io/en/latest/supported_models/supported_models.html 目前只支持了 mincpm-llama3-v-2_5