[Feature Request] Batch Run for chat with list of inputs

[ArtyomZemlyak]

Is your feature request related to a problem? Please describe. If we need test in batch run promptflow pipeline in chat with multiple conversations between user and assistant, for now we only can use chat_history input for passing some previous history. But we need somehow get this history.

Describe the solution you'd like Add support for list input of each of inputs, like this:

{"Q": ["q1_1", "q1_2"], "P": ["p1_1", "p1_2"]}
{"Q": ["q2_1", "q2_2"], "P": ["p2_1", "p2_2"]}

And if batch run get this file, than it iteratively for each line walk through the each of "zip" of this lists. And "simulate" conversation.

Describe alternatives you've considered Also, its can be done through format like this:

[{"Q": "q1_1", "P": "p1_1"}, {"Q": "q1_2", "P": "p1_2"}]
[{"Q": "q2_1", "P": "p2_1"}, {"Q": "q2_2", "P": "p2_2"}]

[0mza987]

Could you please elaborate why you need the Q&P lists and what does it mean by saying "simulate" the conversation?

For chat history management. there are already several tools customers can utilize like OpenAI Threads, langchain memory, and promptflow has no plan to provide such management as a built-in feature since different customers can have different demands.

[ArtyomZemlyak]

@0mza987 I have tests for models when I need to check how they work with message history. And in my flow there is: RAG, reformulate question - each of them can produce different results. Therefore, it is not an option to somehow enter intermediate results into the dataset in advance. Plus, I would like a single flow, without the need to create a separate flow purely for generating the final response.

And the solution seems to be support for a list of user requests in batch run.

[0mza987]

Hi @ArtyomZemlyak, Sounds like you could pass the list as a json string and process it to a list object in the actual flow node to achieve this?

[ArtyomZemlyak]

@0mza987 Hi! This could be done if there was a “for” operator at the graph level of pf. But for now I cant saw any resolution

[cloga]

Hi @ArtyomZemlyak ,

Your ask is more related to data simulator area, and it seems more related to batch run and evaluation scenario, but it is not suitable to do in prompt flow itself, as it more like an LLM application with dynamic inputs (it can take data from simulator).

Here is the process for your case in my mind, not sure whether this meet your scenario:

1. data simulator

   • raw data
   • simulator method (rule based, llm based)

2. batch run

   • data set (generated from data simulator in batch or real-time pattern, your main ask is more about here)
   • prompt flow

3. prompt flow define (instance level execution logic)

   • data interface
   • step1
   • step2

Currently, we didn't have a clear plan to support data simulator in prompt flow, even adding data simulator support but from reproducibility wise, we would still recommend to generated evaluation data set first and use them to trigger multi-round tests or evaluations, data simulation and batch still need be two. So, maybe have another process out of prompt flow to generate these synthetic data is workaround for you now.

[ArtyomZemlyak]

@cloga https://github.com/microsoft/promptflow/issues/1691 can we inspire from this: https://github.com/langchain-ai/langgraph

[0mza987]

Seems you have the alternative solution that is introduced in #1691 so I would like close this issue. Thanks for your feedback.