[Bug] ollama 支持 vision的大模型（如minicpm-v）依然存在无法发送图片的情况

[tyzero]

📦 Environment

Docker

📌 Version

v1.17.7

💻 Operating System

Windows

🌐 Browser

Chrome

🐛 Bug Description

minicpm-v 默认配置下（点击重置配置），可以正常发送图片。

但是，经过 更新模型列表（点击获取模型列表）操作后，vision 配置似乎被覆盖删除，

导致模型在UI界面里无法发送图片

📷 Recurrence Steps

No response

🚦 Expected Behavior

No response

📝 Additional Information

No response

[lobehubbot]

👀 @tyzero

Thank you for raising an issue. We will investigate into the matter and get back to you as soon as possible. Please make sure you have given us as much context as possible.\ 非常感谢您提交 issue。我们会尽快调查此事，并尽快回复您。 请确保您已经提供了尽可能多的背景信息。

[dosubot[bot]]

@tyzero 你好！我是Dosu，一个可以帮助你解决 bug、回答问题并成为贡献者的机器人。在等待人类维护者的过程中，我将会帮助你解决问题。

I found a similar closed issue where the vision capability for models was not functioning correctly. The issue was resolved by configuring the model with the appropriate environment variables or model panel settings ^[1]. Additionally, another related issue was fixed by a pull request that enabled vision for the Ollama LLaVA models ^[2].

In your case, it seems that the vision configuration for the minicpm-v model is being overwritten when you update the model list. The Update Model List functionality modifies the configurations related to the models available for chat, including their capabilities like vision. This is handled by the parseModelString and transformToChatModelCards functions ^[3].

To ensure the vision capability remains enabled, you might need to verify and adjust the model configuration after updating the model list. Specifically, check the src/config/modelProviders/ollama.ts file to ensure the vision: true property is set for the minicpm-v model ^[4].

If the issue persists, you may need to manually re-enable the vision capability in the model configuration after updating the model list.

To continue talking to Dosu, mention @dosu.

[tyzero]

@Dosu 谢谢你AI侠，我看了下代码，parseModelString 似乎只针对 在服务端配置了Model List 的情况，才 能够正确解析 模型的附带能力（vision），如果没有配置 modle list，transformToChatModelCards 会使用默认模型配置，这个defaultChatModels应该有问题。我用的lobechat 是 其他人部署的，但ollama是我本地电脑的服务，我只是需要用到 lobechat 的UI，但却需要去后台配置它支持的能力，这很不合理。

[lobehubbot]

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

---

@Dosu Thank you AI Xia. I looked at the code. It seems that parseModelString can only correctly parse the model's accompanying capabilities (vision) when the Model List is configured on the server. If the model list is not configured, transformToChatModelCards will use the default model. Configuration, there should be something wrong with this defaultChatModels. The lobechat I use is deployed by others, but ollama is a service on my local computer. I just need to use the lobechat UI, but I need to configure the capabilities it supports in the background, which is very unreasonable.

[dosubot[bot]]

To ensure the vision capability is enabled for the minicpm-v model when the Model List is not configured on the server, you can modify the defaultChatModels configuration directly. Here's how you can do it:

1. Locate the defaultChatModels Configuration: Ensure that the defaultChatModels array includes the minicpm-v model with the vision capability enabled.

2. Modify the Configuration: Update the defaultChatModels array to include the minicpm-v model with the vision capability. Here is an example of how you can do this:

import { transformToChatModelCards } from './src/utils/parseModels.ts';
import { ChatModelCard } from '@/types/llm';

const defaultChatModels: ChatModelCard[] = [
  {
    description: 'MiniCPM-V 是 OpenBMB 推出的新一代多模态大模型，具备卓越的 OCR 识别和多模态理解能力，支持广泛的应用场景。',
    displayName: 'MiniCPM-V 8B',
    enabled: true,
    id: 'minicpm-v',
    tokens: 128_000,
    vision: true,
  },
  // Add other default models here
];

const modelString = 'minicpm-v<vision>';

const updatedChatModels = transformToChatModelCards({
  modelString,
  defaultChatModels,
  withDeploymentName: false,
});

console.log(updatedChatModels);

In this example, the minicpm-v model is explicitly configured with the vision capability enabled. The transformToChatModelCards function will process this configuration and ensure that the minicpm-v model retains its vision capability even when the Model List is not configured on the server ^[1].

By ensuring the defaultChatModels array is correctly configured, you can avoid the need to manually re-enable the vision capability after updating the model list.

To continue talking to Dosu, mention @dosu.

[water110]

@tyzero 你好！我是Dosu，一个可以帮助你解决 bug、回答问题并成为贡献者的机器人。在等待人类维护者的过程中，我将会帮助你解决问题。

I found a similar closed issue where the vision capability for models was not functioning correctly. The issue was resolved by configuring the model with the appropriate environment variables or model panel settings [1]. Additionally, another related issue was fixed by a pull request that enabled vision for the Ollama LLaVA models [2].

In your case, it seems that the vision configuration for the minicpm-v model is being overwritten when you update the model list. The Update Model List functionality modifies the configurations related to the models available for chat, including their capabilities like vision. This is handled by the parseModelString and transformToChatModelCards functions [3].

To ensure the vision capability remains enabled, you might need to verify and adjust the model configuration after updating the model list. Specifically, check the src/config/modelProviders/ollama.ts file to ensure the vision: true property is set for the minicpm-v model [4].

If the issue persists, you may need to manually re-enable the vision capability in the model configuration after updating the model list.

To continue talking to Dosu, mention @dosu.

我是通过docker安装的lobehub,我没有找到这个文件src/config/modelProviders/ollama.ts，请问我应该怎么修改，能否在界面支持自定义，而不是通过修改代码的方式实现。