[BUG] Testing streaming tool with VS Code extension does not serialize chat history correctly

[martinmoldrup]

Describe the bug If a flow is using streaming and the VS Code Extension to "Run it with interactive mode (text only)" is used to test it, it does not serialize properly. It serializes the answer as "<promptflow.tracing._trace.TracedIterator object at 0x0000022B85534520>" into the prompt instead of the string.

How To Reproduce the bug Steps to reproduce the behavior, how frequent can you experience the bug:

1. Make a flow that uses a custom tool that streams the answer for the final completion. Make sure it is running in stream mode and the tool is returning a TracedIterator
2. Use the VS Code Extension for "Run it with interactive mode (text only)" and create a short chat history.
3. The prompt chat history is not serialized correctly when you check it in the trace details.

Expected behavior The answer from the assistant should be serialized into the prompt.

Screenshots See screenshot of issue

Running Information(please complete the following information):

• Promptflow Package Version:
• { "promptflow": "1.14.0", "promptflow-azure": "1.14.0", "promptflow-core": "1.14.0", "promptflow-devkit": "1.14.0", "promptflow-tracing": "1.14.0" }
• Operating System: Windows 11
• Python Version: 3.9.18
• Prompt flow Extension: v1.21.133218292 (pre-release)

Additional context None

[khoangothe]

I got this bug too, but for --interactive mode. It seems that in some case, the output doesn't get Serialized and return TraceIterator, so chat_history won't be able to read the result.

How To Reproduce the bug Run the main example on README: pf flow test --flow ./my_chatbot --interactive The bot won't be able to recall any conversation, because chat_history output are TraceIterator

[khoangothe]

I got this bug too, but for --interactive mode. It seems that in some case, the output doesn't get Serialized and return TraceIterator, so chat_history won't be able to read the result.

How To Reproduce the bug Run the main example on README: pf flow test --flow ./my_chatbot --interactive The bot won't be able to recall any conversation, because chat_history output are TraceIterator

I fix it in interactive mode by changing resolve_generator in src/promptflow-devkit/promptflow/_sdk/_orchestrator/utils.py to resolve Iterator too. I think the webUI also call resolve_generator, so changing it might fix both

[oddrationale]

I fix it in interactive mode by changing resolve_generator in src/promptflow-devkit/promptflow/_sdk/_orchestrator/utils.py to resolve Iterator too. I think the webUI also call resolve_generator, so changing it might fix both

Thanks! This fixed it for me, too! Hopefully your PR gets merged in the next version!

[github-actions[bot]]

Hi, we're sending this friendly reminder because we haven't heard back from you in 30 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 7 days of this comment, the issue will be automatically closed. Thank you!