After using the image recognition function and activating the smart assistant, I am unable to draw. It prompts a function error

[leochen-g]

Self Checks

• [X] I have searched for existing issues search for existing issues, including closed ones.
• [X] I confirm that I am using English to file this report (我已阅读并同意 Language Policy).

Dify version

0.5.1

Cloud or Self Hosted

Cloud

Steps to reproduce

1、choose model gpt-4-vision-preview
2、open ai assistant, add dall-e3 tool 3、input：“帮我绘制一副雪人的场景”

✔️ Expected Behavior

After using the image recognition function, when I activate the intelligent assistant, I am unable to draw and receive an error message indicating a function error.

❌ Actual Behavior

error

[openai] Bad Request Error, Error code: 400 - {'error': {'message': '1 validation error for Request\nbody -> functions\n  extra fields not permitted (type=value_error.extra)', 'type': 'invalid_request_error', 'param': None, 'code': None}}

However, if you switch the model to another conversational model, such as gpt-3.5-turbo-16k, you will be able to successfully use the DALL-E3 tool for drawing.

[dosubot[bot]]

🤖

你好，@leochen-g！很高兴再次看到你。希望你一切都好！

根据您提供的信息，我注意到您在使用gpt-4-vision-preview模型与dall-e3工具进行集成时遇到了问题。在当前的dify代码库中，gpt-4-vision-preview模型是被支持的，这可以从YAML配置文件中看出。

然而，关于gpt-4-vision-preview模型是否可以与dall-e3工具进行集成，以及如何进行集成的具体信息，我在代码库中并未找到。这可能是因为这两者的集成需要特定的配置或设置，而这些信息并未在代码库中明确提供。

您遇到的错误提示extra fields not permitted可能是由于在请求中包含了不被gpt-4-vision-preview模型支持的字段。这可能是由于gpt-4-vision-preview模型与dall-e3工具在参数或字段上存在不兼容，或者是在调用过程中使用了错误的参数或字段。

如果可能的话，您能否提供更多关于您如何配置和使用gpt-4-vision-preview模型与dall-e3工具的信息？这将有助于我更好地理解和解决您遇到的问题。

Sources

• api/core/model_runtime/model_providers/openai/llm/gpt-4-vision-preview.yaml

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[Yeuoly]

Function call is not supported by gpt-4-vision-preview currently, and multimodal models will be converted into Tools recently, Thanks for your report.

[leochen-g]

Function call is not supported by gpt-4-vision-preview currently, and multimodal models will be converted into Tools recently, Thanks for your report.

Thanks