swift框架多模态不兼容openai标准接口

Describe the bug What the bug is, and how to reproduce, better with screenshots(描述bug以及复现过程，最好有截图) swift框架多模态不兼容openai标准接口， from openai import OpenAI

client = OpenAI(api_key='YOUR_API_KEY', base_url='http://0.0.0.0:23333/v1') model_name = client.models.list().data[0].id response = client.chat.completions.create( model="InternVL2-8B", messages=[{ 'role': 'user', 'content': [{ 'type': 'text', 'text': '请提取这张照片的内容，其中内容格式‘机器编号’、‘发票代码’、‘发票号码’、‘开票日期’、‘校验码’、‘购买方名称’、‘购买方纳税人识别号’、‘购买方地址、电话’、‘开户行及账号’、‘货物或应税劳务、服务名称’、‘规格型号’、‘单位’、‘数量’、‘单价’、‘金额’、‘税率’、‘税额’、‘价税合计（大写）’、‘价税合计（小写）’、‘销售方名称’、‘销售方纳税人识别号’、‘销售方地址、电话’、‘销售方地址、电话’、‘开户行及账号’、‘备注’、‘收款人’、‘复核’、‘开票人’ 字段返回信息，返回的结果信息以json格式返回', }, { 'type': 'image_url', 'image_url': { 'url': '/tmp/code/22.png', }, }], }], temperature=0.8, top_p=0.8) print(response)

这个使用第三方客户端软件比如chatbox 使用兼容openai 方式调用swift 提供的对外接口服务使用不了。是否能够改进适配兼容openai标准接口

多模态模型部署时，传入图片会直接报错，但只传文字就没问题。

好的我修复一下这个问题

大佬啥时候修复，坐等修复。

目前合并lora权重后部署没有问题了，但是如果--ckpt_dir传入的是lora权重地址，会报错，错误详情如下：

INFO: 2024-07-12 08:57:13,950 deploy.py:378] {'request_id': 'chatcmpl-3e812edccde244dbb96dfdad51718ad5', 'model': 'internvl-chat-v1_5', 'messages': [{'role': 'user', 'content': 'Picture 1:\n图中是什么花，有几只？'}], 'generation_config': GenerationConfig({'do_sample': True, 'max_new_tokens': 30944, 'temperature': 0.3, 'top_k': 20, 'top_p': 0.7}), 'seed': 42, 'stop': [], 'stream': False}
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/miniconda3/envs/swift/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/swift/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/swift/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/home/miniconda3/envs/swift/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/miniconda3/envs/swift/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/home/miniconda3/envs/swift/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/home/miniconda3/envs/swift/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/home/miniconda3/envs/swift/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/home/miniconda3/envs/swift/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/home/miniconda3/envs/swift/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/miniconda3/envs/swift/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/home/miniconda3/envs/swift/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/home/miniconda3/envs/swift/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/home/miniconda3/envs/swift/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/home/miniconda3/envs/swift/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/home/miniconda3/envs/swift/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/swift/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/swift/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/swift/swift/llm/deploy.py", line 507, in create_chat_completion
    return await inference_pt_async(request, raw_request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/swift/swift/llm/deploy.py", line 495, in inference_pt_async
    return await _generate_full()
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/swift/swift/llm/deploy.py", line 396, in _generate_full
    response, _ = inference(
                  ^^^^^^^^^^
  File "/home/miniconda3/envs/swift/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/swift/swift/llm/utils/utils.py", line 800, in inference
    generate_ids = model.generate(streamer=streamer, generation_config=generation_config, **inputs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/swift/lib/python3.11/site-packages/peft/peft_model.py", line 1491, in generate
    outputs = self.base_model.generate(*args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/swift/swift/llm/utils/model.py", line 3764, in _new_generate
    return generate(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/swift/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/.cache/huggingface/modules/transformers_modules/InternVL-Chat-V1-5/modeling_internvl_chat.py", line 344, in generate
    vit_embeds = self.extract_feature(pixel_values)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/swift/swift/llm/utils/model.py", line 3774, in _new_extract_feature
    return extract_feature(pixel_values).to(pixel_values.device).to(pixel_values.dtype)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/.cache/huggingface/modules/transformers_modules/InternVL-Chat-V1-5/modeling_internvl_chat.py", line 250, in extract_feature
    vit_embeds = self.vision_model(
                 ^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/swift/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/swift/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/swift/lib/python3.11/site-packages/accelerate/hooks.py", line 166, in new_forward
    output = module._old_forward(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/.cache/huggingface/modules/transformers_modules/InternVL-Chat-V1-5/modeling_intern_vit.py", line 411, in forward
    encoder_outputs = self.encoder(
                      ^^^^^^^^^^^^^
  File "/home/miniconda3/envs/swift/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/swift/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/swift/lib/python3.11/site-packages/accelerate/hooks.py", line 166, in new_forward
    output = module._old_forward(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/.cache/huggingface/modules/transformers_modules/InternVL-Chat-V1-5/modeling_intern_vit.py", line 347, in forward
    layer_outputs = encoder_layer(
                    ^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/swift/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/swift/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/swift/lib/python3.11/site-packages/accelerate/hooks.py", line 166, in new_forward
    output = module._old_forward(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/.cache/huggingface/modules/transformers_modules/InternVL-Chat-V1-5/modeling_intern_vit.py", line 289, in forward
    hidden_states = hidden_states + self.drop_path1(self.attn(self.norm1(hidden_states)) * self.ls1)
                                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/swift/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/swift/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/swift/lib/python3.11/site-packages/accelerate/hooks.py", line 166, in new_forward
    output = module._old_forward(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/.cache/huggingface/modules/transformers_modules/InternVL-Chat-V1-5/modeling_intern_vit.py", line 246, in forward
    x = self._naive_attn(hidden_states) if not self.use_flash_attn else self._flash_attn(hidden_states)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/.cache/huggingface/modules/transformers_modules/InternVL-Chat-V1-5/modeling_intern_vit.py", line 211, in _naive_attn
    qkv = self.qkv(x).reshape(B, N, 3, self.num_heads, C // self.num_heads).permute(2, 0, 3, 1, 4)
          ^^^^^^^^^^^
  File "/home/miniconda3/envs/swift/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/swift/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1582, in _call_impl
    result = forward_call(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/envs/swift/lib/python3.11/site-packages/peft/tuners/lora/layer.py", line 545, in forward
    self._check_forward_args(x, *args, **kwargs)
  File "/home/miniconda3/envs/swift/lib/python3.11/site-packages/peft/tuners/lora/layer.py", line 332, in _check_forward_args
    raise ValueError(msg)
ValueError: Length of `adapter_names` should be the same as the number of inputs, but got 1 and 7 respectively.

应该是部分修复了，还是有些BUG. 我用上角的测试脚本测试是通过的，因为上面是不是流式输出。今天试用使用第三方chatbox 调用测试后端报错 INFO: 127.0.0.1:33750 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error ERROR: Exception in ASGI application Traceback (most recent call last): File "/usr/local/corex-4.0.0.20240614.114/lib64/python3/dist-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi result = await app( # type: ignore[func-returns-value] File "/usr/local/corex-4.0.0.20240614.114/lib64/python3/dist-packages/uvicorn/middleware/proxy_headers.py", line 70, in call return await self.app(scope, receive, send) File "/usr/local/corex-4.0.0.20240614.114/lib64/python3/dist-packages/fastapi/applications.py", line 1054, in call await super().call(scope, receive, send) File "/usr/local/corex-4.0.0.20240614.114/lib64/python3/dist-packages/starlette/applications.py", line 123, in call await self.middleware_stack(scope, receive, send) File "/usr/local/corex-4.0.0.20240614.114/lib64/python3/dist-packages/starlette/middleware/errors.py", line 186, in call raise exc File "/usr/local/corex-4.0.0.20240614.114/lib64/python3/dist-packages/starlette/middleware/errors.py", line 164, in call await self.app(scope, receive, _send) File "/usr/local/corex-4.0.0.20240614.114/lib64/python3/dist-packages/starlette/middleware/exceptions.py", line 65, in call await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "/usr/local/corex-4.0.0.20240614.114/lib64/python3/dist-packages/starlette/_exception_handler.py", line 64, in wrapped_app raise exc File "/usr/local/corex-4.0.0.20240614.114/lib64/python3/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "/usr/local/corex-4.0.0.20240614.114/lib64/python3/dist-packages/starlette/routing.py", line 756, in call await self.middleware_stack(scope, receive, send) File "/usr/local/corex-4.0.0.20240614.114/lib64/python3/dist-packages/starlette/routing.py", line 776, in app await route.handle(scope, receive, send) File "/usr/local/corex-4.0.0.20240614.114/lib64/python3/dist-packages/starlette/routing.py", line 297, in handle await self.app(scope, receive, send) File "/usr/local/corex-4.0.0.20240614.114/lib64/python3/dist-packages/starlette/routing.py", line 77, in app await wrap_app_handling_exceptions(app, request)(scope, receive, send) File "/usr/local/corex-4.0.0.20240614.114/lib64/python3/dist-packages/starlette/_exception_handler.py", line 64, in wrapped_app raise exc File "/usr/local/corex-4.0.0.20240614.114/lib64/python3/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "/usr/local/corex-4.0.0.20240614.114/lib64/python3/dist-packages/starlette/routing.py", line 72, in app response = await func(request) File "/usr/local/corex-4.0.0.20240614.114/lib64/python3/dist-packages/fastapi/routing.py", line 278, in app raw_response = await run_endpoint_function( File "/usr/local/corex-4.0.0.20240614.114/lib64/python3/dist-packages/fastapi/routing.py", line 191, in run_endpoint_function return await dependant.call(**values) File "/home/swift/swift/llm/deploy.py", line 507, in create_chat_completion return await inference_pt_async(request, raw_request) File "/home/swift/swift/llm/deploy.py", line 347, in inference_pt_async result = await _prepare_request(request) File "/home/swift/swift/llm/deploy.py", line 120, in _prepare_request messages_join_observation(messages) File "/home/swift/swift/llm/utils/utils.py", line 910, in messages_join_observation 'content'].endswith('Observation:'): AttributeError: 'list' object has no attribute 'endswith'

我再检查一下稍等

应该是部分修复了，还是有些BUG. 我用上角的测试脚本测试是通过的，因为上面是不是流式输出。今天试用使用第三方chatbox 调用测试后端报错 INFO: 127.0.0.1:33750 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error ERROR: Exception in ASGI application Traceback (most recent call last): File "/usr/local/corex-4.0.0.20240614.114/lib64/python3/dist-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi result = await app( # type: ignore[func-returns-value] File "/usr/local/corex-4.0.0.20240614.114/lib64/python3/dist-packages/uvicorn/middleware/proxy_headers.py", line 70, in call return await self.app(scope, receive, send) File "/usr/local/corex-4.0.0.20240614.114/lib64/python3/dist-packages/fastapi/applications.py", line 1054, in call await super().call(scope, receive, send) File "/usr/local/corex-4.0.0.20240614.114/lib64/python3/dist-packages/starlette/applications.py", line 123, in call await self.middleware_stack(scope, receive, send) File "/usr/local/corex-4.0.0.20240614.114/lib64/python3/dist-packages/starlette/middleware/errors.py", line 186, in call raise exc File "/usr/local/corex-4.0.0.20240614.114/lib64/python3/dist-packages/starlette/middleware/errors.py", line 164, in call await self.app(scope, receive, _send) File "/usr/local/corex-4.0.0.20240614.114/lib64/python3/dist-packages/starlette/middleware/exceptions.py", line 65, in call await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "/usr/local/corex-4.0.0.20240614.114/lib64/python3/dist-packages/starlette/_exception_handler.py", line 64, in wrapped_app raise exc File "/usr/local/corex-4.0.0.20240614.114/lib64/python3/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "/usr/local/corex-4.0.0.20240614.114/lib64/python3/dist-packages/starlette/routing.py", line 756, in call await self.middleware_stack(scope, receive, send) File "/usr/local/corex-4.0.0.20240614.114/lib64/python3/dist-packages/starlette/routing.py", line 776, in app await route.handle(scope, receive, send) File "/usr/local/corex-4.0.0.20240614.114/lib64/python3/dist-packages/starlette/routing.py", line 297, in handle await self.app(scope, receive, send) File "/usr/local/corex-4.0.0.20240614.114/lib64/python3/dist-packages/starlette/routing.py", line 77, in app await wrap_app_handling_exceptions(app, request)(scope, receive, send) File "/usr/local/corex-4.0.0.20240614.114/lib64/python3/dist-packages/starlette/_exception_handler.py", line 64, in wrapped_app raise exc File "/usr/local/corex-4.0.0.20240614.114/lib64/python3/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "/usr/local/corex-4.0.0.20240614.114/lib64/python3/dist-packages/starlette/routing.py", line 72, in app response = await func(request) File "/usr/local/corex-4.0.0.20240614.114/lib64/python3/dist-packages/fastapi/routing.py", line 278, in app raw_response = await run_endpoint_function( File "/usr/local/corex-4.0.0.20240614.114/lib64/python3/dist-packages/fastapi/routing.py", line 191, in run_endpoint_function return await dependant.call(**values) File "/home/swift/swift/llm/deploy.py", line 507, in create_chat_completion return await inference_pt_async(request, raw_request) File "/home/swift/swift/llm/deploy.py", line 347, in inference_pt_async result = await _prepare_request(request) File "/home/swift/swift/llm/deploy.py", line 120, in _prepare_request messages_join_observation(messages) File "/home/swift/swift/llm/utils/utils.py", line 910, in messages_join_observation 'content'].endswith('Observation:'): AttributeError: 'list' object has no attribute 'endswith'

chatbox 调用测试中的messages信息可以看一下不

恩中午刚才测试试过了，可以了。之前测试的是顺序我们没讲清楚，就是先文本聊天，然后在实现多模态聊天。这样组合的时候会出问题。上午的版本中单轮对话是不会出错的。但是实际很多第三方客户端工具会把历史聊天信息发给后端模型，这样就会出现组合情况（历史上下文的问题）。现在这个问题应该是修复了，感谢作者，你们的效率非常高

modelscope / ms-swift

swift框架多模态不兼容openai标准接口 #1348