microsoft / promptflow

Build high-quality LLM apps - from prototyping, testing to production deployment and monitoring.
https://microsoft.github.io/promptflow/
MIT License
9.42k stars 855 forks source link

[Feature Request] Batch Run for chat with list of inputs #1389

Closed ArtyomZemlyak closed 9 months ago

ArtyomZemlyak commented 11 months ago

Is your feature request related to a problem? Please describe. If we need test in batch run promptflow pipeline in chat with multiple conversations between user and assistant, for now we only can use chat_history input for passing some previous history. But we need somehow get this history.

Describe the solution you'd like Add support for list input of each of inputs, like this:

{"Q": ["q1_1", "q1_2"], "P": ["p1_1", "p1_2"]}
{"Q": ["q2_1", "q2_2"], "P": ["p2_1", "p2_2"]}

And if batch run get this file, than it iteratively for each line walk through the each of "zip" of this lists. And "simulate" conversation.

Describe alternatives you've considered Also, its can be done through format like this:

[{"Q": "q1_1", "P": "p1_1"}, {"Q": "q1_2", "P": "p1_2"}]
[{"Q": "q2_1", "P": "p2_1"}, {"Q": "q2_2", "P": "p2_2"}]
0mza987 commented 10 months ago

Could you please elaborate why you need the Q&P lists and what does it mean by saying "simulate" the conversation?

For chat history management. there are already several tools customers can utilize like OpenAI Threads, langchain memory, and promptflow has no plan to provide such management as a built-in feature since different customers can have different demands.

ArtyomZemlyak commented 10 months ago

@0mza987 I have tests for models when I need to check how they work with message history. And in my flow there is: RAG, reformulate question - each of them can produce different results. Therefore, it is not an option to somehow enter intermediate results into the dataset in advance. Plus, I would like a single flow, without the need to create a separate flow purely for generating the final response.

And the solution seems to be support for a list of user requests in batch run.

0mza987 commented 10 months ago

Hi @ArtyomZemlyak, Sounds like you could pass the list as a json string and process it to a list object in the actual flow node to achieve this?

ArtyomZemlyak commented 10 months ago

@0mza987 Hi! This could be done if there was a “for” operator at the graph level of pf. But for now I cant saw any resolution

cloga commented 10 months ago

Hi @ArtyomZemlyak ,

Your ask is more related to data simulator area, and it seems more related to batch run and evaluation scenario, but it is not suitable to do in prompt flow itself, as it more like an LLM application with dynamic inputs (it can take data from simulator).

Here is the process for your case in my mind, not sure whether this meet your scenario:

  1. data simulator

    • raw data
    • simulator method (rule based, llm based)
  2. batch run

    • data set (generated from data simulator in batch or real-time pattern, your main ask is more about here)
    • prompt flow
  3. prompt flow define (instance level execution logic)

    • data interface
    • step1
    • step2

Currently, we didn't have a clear plan to support data simulator in prompt flow, even adding data simulator support but from reproducibility wise, we would still recommend to generated evaluation data set first and use them to trigger multi-round tests or evaluations, data simulation and batch still need be two. So, maybe have another process out of prompt flow to generate these synthetic data is workaround for you now.

ArtyomZemlyak commented 9 months ago

@cloga https://github.com/microsoft/promptflow/issues/1691 can we inspire from this: https://github.com/langchain-ai/langgraph

0mza987 commented 9 months ago

Seems you have the alternative solution that is introduced in #1691 so I would like close this issue. Thanks for your feedback.