tongyi model code block mode wrapper is not stable

Self Checks

[X] This is only for bug report, if you would like to ask a question, please head to Discussions.
[X] I have searched for existing issues search for existing issues, including closed ones.
[X] I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
[X] [FOR CHINESE USERS] 请务必使用英文提交 Issue，否则会被关闭。谢谢！:）
[X] Please do not modify this template :) and fill in all the required fields.

Dify version

0.7.2

Cloud or Self Hosted

Self Hosted (Docker)

Steps to reproduce

https://github.com/langgenius/dify/pull/5396 This mr try to resolve this problem, but it works not stable because the llm's answer is not stable. Try to ask the same question for some times in that issue, and you may get empty string in some case.

In fact, the basic class already provides a solution with good compatibility, compatible with markdown and non-markdown format returns.

I'll submit a mr to resolve this problem later.

✔️ Expected Behavior

the json result returns stable with markdown or none-markdown result. the llm output: qwen-json-bug-fix-2 the result:

❌ Actual Behavior

the output of llm is not stable, and the json result may be empty string qwen-json-bug2 get some empty string

@zouchengzhuo Hello! I'm here to help you with any bugs, questions, or contributions you have.

To stabilize the JSON output from the Tongyi model code block mode wrapper in Dify version 0.7.2 when using a self-hosted Docker setup, ensure that the response_format parameter is correctly set to "JSON" and that the stop words are properly configured to handle the JSON output.

Here is the relevant part of the _code_block_mode_wrapper method that you need to focus on:

def _code_block_mode_wrapper(self, model: str, credentials: dict,
                             prompt_messages: list[PromptMessage], model_parameters: dict,
                             tools: list[PromptMessageTool] | None = None, stop: list[str] | None = None,
                             stream: bool = True, user: str | None = None, callbacks: list[Callback] = None) \
        -> LLMResult | Generator:
    """
    Wrapper for code block mode
    """
    block_prompts = """You should always follow the instructions and output a valid {{block}} object.
The structure of the {{block}} object you can found in the instructions, use {"answer": "$your_answer"} as the default structure
if you are not sure about the structure.

<instructions>
{{instructions}}
</instructions>
You should also complete the text started with ``` but not tell ``` directly.
"""
    code_block = model_parameters.get("response_format", "")
    if not code_block:
        return self._invoke(
            model=model,
            credentials=credentials,
            prompt_messages=prompt_messages,
            model_parameters=model_parameters,
            tools=tools,
            stop=stop,
            stream=stream,
            user=user
        )

    model_parameters.pop("response_format")
    stop = stop or []
    stop.extend(["\n```", "```\n"])
    block_prompts = block_prompts.replace("{{block}}", code_block)

    # check if there is a system message
    if len(prompt_messages) > 0 and isinstance(prompt_messages[0], SystemPromptMessage):
        # override the system message
        prompt_messages[0] = SystemPromptMessage(
            content=block_prompts
            .replace("{{instructions}}", prompt_messages[0].content)
        )
    else:
        # insert the system message
        prompt_messages.insert(0, SystemPromptMessage(
            content=block_prompts
            .replace("{{instructions}}", f"Please output a valid {code_block} with markdown codeblocks.")
        ))

    if len(prompt_messages) > 0 and isinstance(prompt_messages[-1], UserPromptMessage):
        # add ```JSON\n to the last message
        prompt_messages[-1].content += f"\n```{code_block}\n"
    else:
        # append a user message
        prompt_messages.append(UserPromptMessage(
            content=f"```{code_block}\n"
        ))

    response = self._invoke(
        model=model,
        credentials=credentials,
        prompt_messages=prompt_messages,
        model_parameters=model_parameters,
        tools=tools,
        stop=stop,
        stream=stream,
        user=user
    )

Ensure that the response_format in model_parameters is set to "JSON" and that the stop words include \n````, and ``\n to properly handle the JSON output. This will help in stabilizing the JSON output from the Tongyi model ^[1].

To continue talking to Dosu, mention @dosu.

langgenius / dify