dosubot[bot]
commented
5 months ago 🤖
Thank you for your feature request and your interest in contributing to the dify project. Your suggestion to allow large language models to provide outputs in JSON format using a Pydantic parser is indeed a valuable one. This feature would indeed improve the integration of dify with other systems.
The current implementation of the dify platform handles the responses from the large language model in the _handle_generate_response and _handle_generate_stream_response methods in the OAIAPICompatLargeLanguageModel class. These methods return the responses as instances of the LLMResult or LLMResultChunk classes, which are defined in the core.model_runtime.entities.llm_entities module.
To implement your feature, we would need to modify these methods to return the responses in a JSON format. This could be achieved by using the json module's dumps function to convert the LLMResult or LLMResultChunk instances to JSON strings. However, since these classes may contain non-serializable types, we may need to define custom serialization methods for them.
To use a Pydantic parser, we could define Pydantic models that mirror the structure of the LLMResult and LLMResultChunk classes, and use the parse_obj method of these Pydantic models to parse the JSON strings.
Here is an example of how we could modify the _handle_generate_response method to return the response in a JSON format:
from pydantic import BaseModel
import json
class LLMResultPydantic(BaseModel):
model: str
prompt_messages: list[PromptMessage]
message: AssistantPromptMessage
usage: dict
# Define any other fields that are present in the LLMResult class
def _handle_generate_response(self, model: str, credentials: dict, response: requests.Response,
prompt_messages: list[PromptMessage]) -> str:
# Existing code to generate the LLMResult instance
# ...
result = LLMResult(
model=response_json["model"],
prompt_messages=prompt_messages,
message=assistant_message,
usage=usage,
)
# Convert the LLMResult instance to a JSON string
result_json = json.dumps(result.__dict__)
# Parse the JSON string using the Pydantic model
result_pydantic = LLMResultPydantic.parse_raw(result_json)
return result_pydantic.json()Please note that this is a simplified example and you may need to adjust it according to the actual structure of the LLMResult and LLMResultChunk classes and the types of their fields.
You can find the current structure of the LLMResult and LLMResultChunk classes in the core.model_runtime.entities.llm_entities module in the dify repository.
I hope this information is helpful. If you have any further questions or need more clarification, please don't hesitate to ask.
Sources
- api/core/model_runtime/model_providers/openai_api_compatible/llm/llm.py
- api/core/model_runtime/entities/llm_entities.py
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.
Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.
If you want to continue the conversation, start your reply with @dosu-bot.
badbye
takatost
1. Is this request related to a challenge you're experiencing?
Yes. For optimal integration with other systems, it's crucial for large models to provide outputs in JSON format.
2. Describe the feature you'd like to see
Specify the output format on the webpage, the server will transform it into a Pydantic model. Utilize Pydantic parser to add instructions into the prompt, and interpret the results into a structured output.
3. How will this feature improve your workflow or experience?
Via API requests, it delivers a preset JSON format that can be seamlessly integrated with any other system.
4. Additional context or comments
No response
5. Can you help us with this feature?