Closed KatoStevenMubiru closed 3 weeks ago
Hi @KatoStevenMubiru
Thank you for your contribution request. I have a few questions:
Hi @0mza987,
Thanks for your questions. Here's some clarification:
Unify (https://unify.ai/docs/index.html) is an open-source framework for unified access to various LLMs across providers, offering optimization for quality, speed, and cost-efficiency.
I'm proposing either a community integration or a package to integrate Unify's capabilities with Promptflow, not a specific flow example. This could involve custom components utilizing Unify's features in Promptflow workflows.
Which approach - community integration or package - would better align with Promptflow's architecture and plans?
Let me know if you need any further information.
@KatoStevenMubiru Thanks for the clarification.
Could you please provide a pseudo-code example to demonstrate how users would utilize Promptflow along with the integrated Unify features? This would help us better understand the targeted user scenario.
My initial thought is that a community integration (which, if I understand correctly, involves adding Unify to Promptflow's dependencies) may not be feasible unless it addresses a significant user need and Unify is proved to be the most suitable tool for this purpose. At this stage, developing a new package might be a more practical solution.
cc @eniac871 @wangchao1230
Thank you for your feedback and questions, @0mza987. I understand your concerns about community integration. Let me provide a pseudo-code example that demonstrates how users could leverage Promptflow with integrated Unify features, addressing significant user needs:
from promptflow import flow, tool
from unify import Unify
class UnifyTool:
def __init__(self):
self.unify_client = Unify(api_key="YOUR_API_KEY")
@tool
def optimize_llm(self, prompt: str, constraints: dict):
optimal_model = self.unify_client.select_optimal_model(constraints)
response = self.unify_client.generate(prompt, model=optimal_model)
return response
@tool
def benchmark_models(self, prompt_set: list, models: list):
results = self.unify_client.benchmark(prompt_set, models)
return results
@flow
def adaptive_translation_flow(text: str, target_language: str):
unify_tool = UnifyTool()
# Step 1: Determine text complexity
complexity_prompt = f"Analyze the complexity of this text: {text}"
complexity_result = unify_tool.optimize_llm(
complexity_prompt,
constraints={"max_latency": 1000, "min_quality": 0.7}
)
# Step 2: Choose appropriate model based on complexity
if "high complexity" in complexity_result.lower():
translation_constraints = {"min_quality": 0.9, "max_cost": 0.05}
else:
translation_constraints = {"min_quality": 0.7, "max_cost": 0.02}
# Step 3: Perform translation
translation_prompt = f"Translate the following text to {target_language}: {text}"
translation = unify_tool.optimize_llm(translation_prompt, constraints=translation_constraints)
# Step 4: Quality check
quality_check_prompt = f"Evaluate the quality of this translation: Original: {text}, Translation: {translation}"
quality_score = unify_tool.optimize_llm(
quality_check_prompt,
constraints={"max_latency": 1500, "min_quality": 0.8}
)
return {
"original_text": text,
"translated_text": translation,
"quality_score": quality_score
}
# Usage
result = adaptive_translation_flow("Hello, how are you?", "French")
print(result)
# Benchmarking
benchmark_results = UnifyTool().benchmark_models(
["Translate 'Hello' to Spanish", "Translate 'Goodbye' to German"],
["gpt-3.5-turbo", "gpt-4", "claude-2"]
)
print(benchmark_results)
This example demonstrates how Unify could be integrated into Promptflow to address significant user needs:
Adaptive Model Selection: The flow dynamically selects the most appropriate LLM based on text complexity and user-defined constraints (latency, quality, cost).
Multi-step Workflows: It showcases a multi-step translation process, leveraging different models for complexity analysis, translation, and quality checking.
Performance Optimization: By using Unify's optimize_llm
tool, the flow ensures each step uses the best-performing model within given constraints.
Benchmarking: Users can easily benchmark multiple models for specific tasks, helping them make informed decisions about which models to use in their flows.
Cost Management: The flow demonstrates how to adjust model selection based on task requirements, potentially reducing costs for simpler tasks.
Quality Assurance: It includes a built-in quality check step, ensuring the output meets the required standards.
Benefits for Promptflow users:
I'm happy to provide additional examples showcasing other Unify features that could benefit Promptflow users. Some areas we could explore further include:
If any of these or other aspects interest the Promptflow team, I can prepare more detailed pseudo-code examples. Our aim is to demonstrate how Unify can enhance Promptflow's capabilities and address user needs.
Hi @KatoStevenMubiru,
Thank you for the detailed explanation and code example. Based on the sample code, it seems that the Unify features can be seamlessly integrated into our flex flow. Please review the following guidelines and examples and see if it can meet your needs:
Hi @0mza987 ,
Following suggestions from my superiors and our previous discussions, we propose adding a unify.py
module to the tools directory of Promptflow.
This file will integrate Unify’s functionalities, enabling Promptflow to access a diverse range of LLMs and optimize model selection based on latency, quality, and cost criteria. Our goal is to enhance the flexibility and efficiency of Promptflow through this integration.
Looking forward to your thoughts and any additional requirements you might have.
Where by we aim to add the Unify integration in the section for Connections in Promptflow documentation. This will enhance Promptflow's capabilities by allowing access to a diverse range of LLMs with optimized model selection based on latency, quality, and cost.
Hi @KatoStevenMubiru, thanks for reaching out! In addition to flex flow, you can also consider to make it a custom tool following this guidance. Then you can submit a PR to this repo to contribute a tool doc and contribute a sample flow. For example: a tool doc, a sample flow for tool. Please feel free to contact me if you need any help on the contribution process. :)
Okay thank you both @ChenJieting and @0mza987 for the help. Lets make an initial implementation and submit a pull request.
@KatoStevenMubiru Just to clarify, now there are 3 different options to integrate Unify features according to above discussion:
unify.py
module in the tools directly: This is not recommended. Tools directory only contains built-in tools, all 3rd party tools need to be costum tools. Not to mention this will add unify package as a dependency to prompt flow.You can choose the implementation 1 or 2. Please make sure you are not implementing method 3, otherwise we may not be able to merge it.
Thanks.
Okay @0mza987 , the last unify.py
option, is out , we will focus on the first 2 options.
Thank you
Hi @0mza987 and @ChenJieting,
I wanted to share a suggested integration plan i've outlined for bringing Unify into Promptflow, complete with specific file names. Could you please confirm if this aligns with your expectations?
unify_llm_tool.py.
test_unify_llm_tool.py
to ensure comprehensive testing.README.md
and add detailed docstrings in unify_llm_tool.py.
flow.dag.yaml
and optimize_llm.py.
flake8
and black.
Project structure:
unify-promptflow-integration/
│
├── src/
│ └── unify_llm_tool/
│ ├── __init__.py
│ ├── unify_llm_tool.py
│ ├── utils.py
│ └── yamls/
│ └── unify_llm_tool.yaml
│
├── tests/
│ ├── __init__.py
│ └── test_unify_llm_tool.py
│
├── examples/
│ └── unify_flow_example/
│ ├── flow.dag.yaml
│ └── optimize_llm.py
│
├── docs/
│ └── README.md
│
├── .github/
│ └── workflows/
│ └── ci.yml
│
├── .gitignore
├── setup.py
├── requirements.txt
├── MANIFEST.in
├── LICENSE
└── README.md
Do these steps look good to you, or is there anything else we should consider?
Thanks
Hi @KatoStevenMubiru,
For the tool source code, you can create a new repo on github and develop it like any other project (public or private, both are fine).
Then you can submit a PR to our repo to add tool document
and flow examples
, please refer to:
Hi @KatoStevenMubiru, I agree with @0mza987, for the tool contribution, we'd suggest you submitting a PR to add/update the documentation following the structure we have built: 1) Add a new page for your tool documentation to this directory, for example: azure-ai-language-tool document. 2) Update the custom tool index table by adding your tool package meta info. 3) Add the sample flow for your tool to this directory following the Readme guidance. For example, azure-ai-language-tool sample flows Thank you!
Okay thank you @ChenJieting and @0mza987 , I have made an implementation . I will submit our repo once everything is set.
Thank you, I truly appreciate your advise and time.
I've addressed your comments in PR. Please review at you convenience. @0mza987 @ChenJieting
@RiddhiJivani I have merged your pr, thanks for your effort.
Hello Promptflow team,
I'm Kato Steven Mubiru, an engineer and contributor from Unify. I'll be leading the integration of Unify into Promptflow.
As the lead on this integration effort, my plan is to:
I'm starting work on this today and will provide regular updates on the progress. If there are any specific Promptflow requirements or considerations you'd like me to be aware of, please let me know.
I look forward to collaborating with the Promptflow team to make this integration successful.