使用TurboMind 推理 + Python 代码集成的方式报错

[zhanghanweii]

Checklist

• [X] 1. I have searched related issues but cannot get the expected help.
• [ ] 2. The bug has not been fixed in the latest version.

Describe the bug

Traceback (most recent call last): File "/mnt/data/data/user/zhanghanwei/LLM_model/SpeechGPT-main/speechgpt/src/infer/cli_infer-lmdeploy.py", line 269, in interact self.forward([prompt]) File "/mnt/data/data/user/zhanghanwei/LLM_model/SpeechGPT-main/speechgpt/src/infer/cli_infer-lmdeploy.py", line 186, in forward for a, b in enumerate(outputs): TypeError: 'EngineOutput' object is not iterable 'EngineOutput' object is not iterable

Reproduction

下面是我的代码，参考自（https://xujinzh.github.io/2024/01/13/ai-internlm-lmdeploy/index.html）： for outputs in self.generator.stream_infer( session_id=0, input_ids=[input_ids]): res, tokens = outputs[0]

Environment

Package                       Version               Editable project location
----------------------------- --------------------- ---------------------------------------------------------------------------
absl-py                       2.0.0
accelerate                    0.31.0
adaseq                        0.6.6
addict                        2.4.0
aiofiles                      23.2.1
aiohttp                       3.9.3
aioprometheus                 23.12.0
aiosignal                     1.3.1
albumentations                1.3.1
alias-free-torch              0.0.6
aliyun-python-sdk-core        2.15.1
aliyun-python-sdk-kms         2.16.2
altair                        5.3.0
aniso8601                     9.0.1
annotated-types               0.6.0
antlr4-python3-runtime        4.9.3
anyio                         4.2.0
apex                          0.1
appdirs                       1.4.4
argon2-cffi                   23.1.0
argon2-cffi-bindings          21.2.0
arrow                         1.3.0
asttokens                     2.4.1
astunparse                    1.6.3
async-lru                     2.0.4
async-timeout                 4.0.3
attrs                         23.2.0
audioread                     3.0.1
auto-gptq                     0.6.0
autoawq                       0.2.4
autoawq_kernels               0.0.6
av                            12.0.0
Babel                         2.14.0
basicsr                       1.4.2
beartype                      0.18.5
beautifulsoup4                4.12.3
bidict                        0.23.1
biopython                     1.82
bitarray                      2.9.2
bitsandbytes                  0.42.0
bitstring                     4.2.0
black                         23.12.1
bleach                        6.1.0
blis                          0.7.11
blobfile                      2.1.1
bmt-clipit                    1.0
boltons                       23.0.0
boto3                         1.34.88
botocore                      1.34.88
brotlipy                      0.7.0
cachetools                    5.3.2
catalogue                     2.0.10
certifi                       2023.11.17
cffi                          1.15.1
cfgv                          3.4.0
charset-normalizer            2.0.4
chumpy                        0.70
cityscapesScripts             2.2.3
click                         8.1.7
clip                          1.0
cloudpathlib                  0.16.0
cloudpickle                   3.0.0
colorama                      0.4.6
coloredlogs                   14.0
comm                          0.2.1
conda                         23.9.0
conda-content-trust           0.2.0
conda-libmamba-solver         23.9.1
conda-package-handling        2.2.0
conda_package_streaming       0.9.0
confection                    0.1.4
ConfigArgParse                1.7
contextlib2                   21.6.0
contourpy                     1.2.0
control-ldm                   0.0.1
crcmod                        1.7
cryptography                  41.0.3
cycler                        0.12.1
cymem                         2.0.8
Cython                        0.29.36
dacite                        1.8.1
dataclasses                   0.6
datasets                      2.16.1
ddpm-guided-diffusion         0.0.0
debugpy                       1.8.0
decorator                     4.4.2
decord                        0.6.0
deepspeed                     0.12.6
defusedxml                    0.7.1
descartes                     1.1.0
detectron2                    0.6
dgl                           1.1.3+cu121
diffusers                     0.27.2
dill                          0.3.7
Distance                      0.1.3
distlib                       0.3.8
dnspython                     2.3.0
docstring_parser              0.16
easydict                      1.13
easyrobust                    0.2.4
edit-distance                 1.0.6
editdistance                  0.5.2
einops                        0.7.0
embeddings                    0.0.8
emoji                         2.11.1
espnet-tts-frontend           0.0.3
et-xmlfile                    1.1.0
evaluate                      0.4.2
eventlet                      0.36.1
exceptiongroup                1.2.0
executing                     2.0.1
expecttest                    0.2.1
face-alignment                1.4.1
fairscale                     0.4.13
fairseq                       0.12.2
fastai                        2.7.14
fastapi                       0.109.0
fastcore                      1.5.29
fastdownload                  0.0.7
fastjsonschema                2.19.1
fastprogress                  1.0.3
fasttext                      0.9.2
ffmpeg                        1.4
ffmpeg-python                 0.2.0
ffmpy                         0.3.2
filelock                      3.13.1
fire                          0.6.0
flake8                        7.0.0
flash-attn                    2.5.2
Flask                         2.2.5
Flask-Cors                    4.0.0
Flask-RESTful                 0.3.10
Flask-SocketIO                5.3.6
flask-talisman                1.1.0
flatbuffers                   23.5.26
fonttools                     4.47.0
fqdn                          1.5.1
frozenlist                    1.4.1
fsspec                        2023.10.0
ftfy                          6.2.0
funasr                        1.0.14
funcodec                      0.2.0
funtextprocessing             0.1.1
future                        0.18.3
fvcore                        0.1.5.post20221221
g2p                           2.0.0
g2p-en                        2.1.0
gast                          0.5.4
gekko                         1.0.6
google-auth                   2.26.1
google-auth-oauthlib          1.0.0
google-pasta                  0.2.0
gradio                        3.50.2
gradio_client                 0.6.1
greenlet                      3.0.3
grpcio                        1.60.0
h11                           0.14.0
h5py                          3.10.0
hdbscan                       0.8.33
hjson                         3.1.0
httpcore                      1.0.5
httptools                     0.6.1
httpx                         0.27.0
huggingface-hub               0.20.3
humanfriendly                 10.0
hydra-core                    1.3.2
HyperPyYAML                   1.2.2
identify                      2.5.36
idna                          3.4
imageio                       2.33.1
imageio-ffmpeg                0.4.9
imgaug                        0.4.0
importlib-metadata            7.0.1
importlib_resources           6.4.0
inflect                       7.0.0
iniconfig                     2.0.0
iopath                        0.1.9
ipdb                          0.13.13
ipykernel                     6.28.0
ipython                       8.19.0
isoduration                   20.11.0
isort                         5.13.2
itsdangerous                  2.2.0
jaconv                        0.3.4
jamo                          0.4.1
jedi                          0.19.1
jieba                         0.42.1
Jinja2                        3.1.2
jmespath                      0.10.0
joblib                        1.3.2
json-tricks                   3.17.3
json5                         0.9.14
jsonpatch                     1.32
jsonplus                      0.8.0
jsonpointer                   2.1
jsonschema                    4.21.1
jsonschema-specifications     2023.12.1
jupyter_client                8.6.0
jupyter_core                  5.7.0
jupyter-events                0.9.0
jupyter-lsp                   2.2.2
jupyter_server                2.12.5
jupyter_server_terminals      0.5.2
jupyterlab                    4.0.12
jupyterlab_pygments           0.3.0
jupyterlab_server             2.25.2
kaldiio                       2.18.0
kantts                        1.0.1
keras                         2.14.0
kiwisolver                    1.4.5
kornia                        0.7.2
kornia_rs                     0.1.3
kwsbp                         0.0.6
langcodes                     3.3.0
lap                           0.4.0
lazy_loader                   0.3
libclang                      16.0.6
libmambapy                    1.5.1
librosa                       0.10.1
lightning-utilities           0.11.2
llvmlite                      0.41.1
lmdb                          1.4.1
lmdeploy                      0.4.2                 /mnt/data/data/user/zhanghanwei/LLM_model/SpeechGPT-main/speechgpt/lmdeploy
local-attention               1.9.1
lpips                         0.1.4
lxml                          4.9.4
lyft-dataset-sdk              0.0.8
Markdown                      3.5.1
markdown-it-py                3.0.0
MarkupSafe                    2.1.3
matplotlib                    3.5.3
matplotlib-inline             0.1.6
mccabe                        0.7.0
mdurl                         0.1.2
megatron-util                 1.3.2
MinDAEC                       0.0.2
mir-eval                      0.7
mistune                       3.0.2
ml-collections                0.1.1
ml-dtypes                     0.2.0
mmcls                         0.25.0
mmcv-full                     1.7.0+torch2.1.1cu121
mmdet                         2.28.2
mmdet3d                       1.0.0a1
mmengine-lite                 0.10.4
mmsegmentation                0.30.0
mock                          5.1.0
modelscope                    1.14.0
more-itertools                10.2.0
moviepy                       1.0.3
mpi4py                        3.1.5
mpmath                        1.3.0
ms-swift                      2.0.2
msgpack                       1.0.7
multidict                     6.0.4
multiprocess                  0.70.15
munkres                       1.1.4
murmurhash                    1.0.10
mypy-extensions               1.0.0
nbclient                      0.9.0
nbconvert                     7.14.2
nbformat                      5.9.2
nerfacc                       0.2.2
nest-asyncio                  1.5.8
networkx                      3.2.1
ninja                         1.11.1.1
nltk                          3.8.1
nodeenv                       1.8.0
notebook_shim                 0.2.3
numba                         0.58.1
numpy                         1.26.3
nuscenes-devkit               1.1.11
nvdiffrast                    0.3.1
oauthlib                      3.2.2
omegaconf                     2.3.0
onnx                          1.16.0
onnxruntime                   1.17.3
onnxsim                       0.4.36
open-clip-torch               2.24.0
openai-whisper                20231117
opencv-python                 4.9.0.80
opencv-python-headless        4.9.0.80
openpyxl                      3.1.2
opt-einsum                    3.3.0
optimum                       1.19.1
orjson                        3.9.12
oss2                          2.18.4
overrides                     7.7.0
packaging                     23.1
pai-easycv                    0.11.6
paint-ldm                     0.0.0
pandas                        2.2.0
pandocfilters                 1.5.1
panopticapi                   0.1
panphon                       0.20.0
parso                         0.8.3
pathspec                      0.12.1
peft                          0.4.0
pexpect                       4.9.0
phaseaug                      1.0.1
pickleshare                   0.7.5
pillow                        10.2.0
pip                           24.1
platformdirs                  4.1.0
plotly                        5.21.0
pluggy                        1.5.0
plyfile                       1.0.3
pointnet2                     0.0.0
pooch                         1.8.0
portalocker                   2.8.2
pre-commit                    3.7.0
preshed                       3.0.9
prettytable                   3.10.0
proglog                       0.1.10
prometheus-client             0.19.0
prompt-toolkit                3.0.43
protobuf                      3.20.3
psutil                        5.9.7
ptflops                       0.7.2.2
ptyprocess                    0.7.0
pure-eval                     0.2.2
py-cpuinfo                    9.0.0
py-sound-connect              0.2.1
pyarrow                       15.0.0
pyarrow-hotfix                0.6
pyasn1                        0.5.1
pyasn1-modules                0.3.0
pybind11                      2.11.1
pyclipper                     1.3.0.post5
pycocoevalcap                 1.2
pycocotools                   2.0.7
pycodestyle                   2.11.1
pycosat                       0.6.6
pycparser                     2.21
pycryptodome                  3.20.0
pycryptodomex                 3.20.0
pydantic                      2.5.3
pydantic_core                 2.14.6
pyDeprecate                   0.3.2
pydot                         2.0.0
pydub                         0.25.1
pyflakes                      3.2.0
Pygments                      2.17.2
PyMCubes                      0.1.4
pynini                        2.1.5
pynndescent                   0.5.12
pynvml                        11.5.0
pyOpenSSL                     23.2.0
pyparsing                     3.1.1
pypinyin                      0.44.0
pyquaternion                  0.9.9
PySocks                       1.7.1
pysptk                        0.1.18
pytest                        8.1.1
pythainlp                     5.0.2
python-crfsuite               0.9.10
python-dateutil               2.8.2
python-dotenv                 1.0.1
python-engineio               4.9.0
python-json-logger            2.0.7
python-multipart              0.0.9
python-rapidjson              1.17
python-socketio               5.11.2
pytorch-lightning             1.7.7
pytorch-metric-learning       2.5.0
pytorch-wavelets              1.3.0
pytorch-wpe                   0.0.1
pytorch3d                     0.7.5
pytz                          2023.4
pyvi                          0.1.1
PyWavelets                    1.6.0
PyYAML                        6.0.1
pyzmq                         25.1.2
quantile-python               1.1
qudida                        0.0.4
rapidfuzz                     3.8.1
ray                           2.9.1
referencing                   0.33.0
regex                         2023.12.25
requests                      2.31.0
requests-oauthlib             1.3.1
resampy                       0.4.2
rfc3339-validator             0.1.4
rfc3986-validator             0.1.1
rich                          13.7.1
rotary-embedding-torch        0.5.3
rouge                         1.0.1
rouge-score                   0.0.4
rpds-py                       0.17.1
rsa                           4.9
ruamel.yaml                   0.18.6
ruamel.yaml.clib              0.2.8
ruff                          0.4.9
s3transfer                    0.10.1
sacrebleu                     2.4.0
sacremoses                    0.1.1
safetensors                   0.4.1
scikit-image                  0.19.3
scikit-learn                  1.3.2
scipy                         1.11.4
seaborn                       0.13.2
semantic-version              2.10.0
Send2Trash                    1.8.2
sentencepiece                 0.1.99
seqeval                       1.2.2
setuptools                    68.0.0
Shapely                       1.8.4
shellingham                   1.5.4
shortuuid                     1.0.13
shotdetect-scenedetect-lgss   0.0.4
shtab                         1.7.1
simple-websocket              1.0.0
simplejson                    3.19.2
six                           1.16.0
sklearn-crfsuite              0.3.6
smart-open                    6.4.0
smplx                         0.1.28
sniffio                       1.3.0
sortedcontainers              2.4.0
soundfile                     0.12.1
soupsieve                     2.5
sox                           1.5.0
soxr                          0.3.7
spacy                         3.7.4
spacy-legacy                  3.0.12
spacy-loggers                 1.0.5
speechbrain                   1.0.0
srsly                         2.4.8
sse-starlette                 2.1.0
stack-data                    0.6.3
stanza                        1.8.2
starlette                     0.35.1
subword-nmt                   0.3.8
sympy                         1.12
tabulate                      0.9.0
taming-transformers-rom1504   0.0.6
tenacity                      8.2.3
tensorboard                   2.16.2
tensorboard-data-server       0.7.2
tensorboardX                  2.6.2.2
tensordict                    0.2.1
tensorflow                    2.14.0
tensorflow-estimator          2.14.0
tensorflow-io-gcs-filesystem  0.35.0
termcolor                     2.4.0
terminado                     0.18.0
terminaltables                3.1.10
text-unidecode                1.3
text2sql-lgesql               1.3.0
tf_keras-nightly              2.16.0.dev2024013110
tf-slim                       1.1.0
thinc                         8.2.3
thop                          0.1.1.post2209072238
threadpoolctl                 3.2.0
tifffile                      2024.1.30
tiktoken                      0.5.2
timm                          0.9.16
tinycss2                      1.2.1
tinycudann                    1.7
tokenizers                    0.15.1
toml                          0.10.2
tomli                         2.0.1
tomlkit                       0.12.0
toolz                         0.12.1
torch                         2.1.2+cu121
torch-complex                 0.4.3
torch-scatter                 2.1.2
torchaudio                    2.1.2+cu121
torchdata                     0.7.1
torchmetrics                  0.11.4
torchsde                      0.2.6
torchsummary                  1.5.1
torchvision                   0.16.2+cu121
tornado                       6.4
tqdm                          4.65.0
traitlets                     5.14.1
trampoline                    0.1.2
transformers                  4.38.2
transformers-stream-generator 0.0.4
trimesh                       2.35.39
triton                        2.1.0
tritonclient                  2.46.0
trl                           0.8.6
truststore                    0.8.0
ttsfrd                        0.2.1
typeguard                     2.13.3
typer                         0.12.3
types-python-dateutil         2.8.19.20240106
typing                        3.7.4.3
typing_extensions             4.9.0
tyro                          0.8.3
tzdata                        2023.4
ujson                         5.9.0
umap-learn                    0.5.6
unicodecsv                    0.14.1
unicodedata2                  15.1.0
unicore                       1.2.1
Unidecode                     1.3.8
uri-template                  1.3.0
urllib3                       2.2.1
utils                         1.0.2
uvicorn                       0.27.0.post1
uvloop                        0.19.0
videofeatures-clipit          1.0
virtualenv                    20.25.3
vllm                          0.3.0
wasabi                        1.1.2
watchfiles                    0.21.0
wcwidth                       0.2.12
weasel                        0.3.4
webcolors                     1.13
webencodings                  0.5.1
websocket-client              1.7.0
websockets                    11.0.3
Werkzeug                      3.0.1
wget                          3.2
wheel                         0.41.2
wrapt                         1.14.1
wsproto                       1.2.0
xformers                      0.0.23.post1
xtcocotools                   1.14
xxhash                        3.4.1
yacs                          0.1.8
yapf                          0.30.0
yarl                          1.9.4
zhconv                        1.4.3
zipp                          3.17.0
zstandard                     0.19.0

Error traceback

No response

[zhanghanweii]

我打印了输出如下： EngineOutput(status=<ResponseType.FINISH: 2>, token_ids=[10994, 29991, 1128, 508, 306, 1371, 366, 29973, 29966, 29872, 19807, 29958, 33002, 584, 508, 366, 5649, 825, 319, 29902, 338, 29973, 33004, 33003, 584, 18585, 29991, 319, 29902, 15028, 363, 3012, 928, 616, 3159, 28286, 29892, 607, 338, 263, 1746, 310, 6601, 10466, 393, 8569, 267, 373, 4969, 13052, 296, 14884, 393, 508, 1348, 322, 1044, 763, 25618, 29889, 319, 29902, 6757, 526, 2221, 304, 5110, 515, 1009, 5177, 29892, 18720, 15038, 29892, 322, 1207, 1602, 12112, 2729, 373, 278, 848, 896, 7150, 29889, 319, 29902, 508, 367, 1304, 304, 4505, 4280, 29902, 29973, 33004, 33003, 584, 450, 1900, 982, 304, 5110, 1048, 319, 29902, 338, 304, 1303, 8277, 29892, 2125, 21888, 29892, 322, 14333, 378, 10662, 29889, 19814, 29892, 727, 526, 1784, 7395, 7788, 3625, 29892, 1316, 408, 25410, 29892, 12618, 29879, 29892, 322, 363, 6762, 29889, 739, 338, 884, 4100, 304, 7952, 701, 304, 2635, 411, 278, 9281, 2693, 1860, 297, 319, 29902, 15483, 29889, 9788, 29892, 372, 338, 4100, 304, 6944, 322, 7639, 411, 319, 29902], num_token=513, logprobs=None)

[irexyc]

之前 stream_infer 返回 tuple，现在返回 EngineOutput 结构体。

你可以参考这里 https://github.com/InternLM/lmdeploy/blob/main/lmdeploy/turbomind/chat.py#L127-L142

[zhanghanweii]

之前 stream_infer 返回 tuple，现在返回 EngineOutput 结构体。

你可以参考这里 https://github.com/InternLM/lmdeploy/blob/main/lmdeploy/turbomind/chat.py#L127-L142

谢谢，我在测试的时候发现速度好像并没有比vllm快，我在for outputs in generator.stream_infer()前后添加start=datetime.now()，end=datetime.now()，最后的结果比vllm慢好几倍

[zhanghanweii]

之前 stream_infer 返回 tuple，现在返回 EngineOutput 结构体。

你可以参考这里 https://github.com/InternLM/lmdeploy/blob/main/lmdeploy/turbomind/chat.py#L127-L142

[TM][INFO] ------------------------- step = 580 ------------------------- [TM][INFO] ------------------------- step = 590 ------------------------- [TM][INFO] ------------------------- step = 600 ------------------------- [TM][INFO] [Interrupt] slot = 0, id = 0 [TM][INFO] [forward] Request completed for 0 Hello! How can I help you? text generate cost 0:00:03.319546 耗时是3秒多，vllm只有700ms。我看每次运行都会经过[TM][INFO] ------------------------- step = *** -------------------------这个步骤，这部分目视都花费了几秒

[lvhan028]

怎么测试的呢？确定生成的token数量一样吗？lmdeploy的max_batch_size是多大呢？