microsoft / promptflow

Build high-quality LLM apps - from prototyping, testing to production deployment and monitoring.
https://microsoft.github.io/promptflow/
MIT License
9.49k stars 868 forks source link

[BUG] Flex Flow __aggregate__ method fails silently #3797

Open jomalsan opened 1 month ago

jomalsan commented 1 month ago

Describe the bug When running a flex flow, if an exception occurs in the __aggregate__ method, there will be no logging to the command life when running the flow.

How To Reproduce the bug Steps to reproduce the behavior, how frequent can you experience the bug:

  1. Build a flex flow
  2. Add raise Exception("Test") to the __aggregate__ method
  3. Run the flow

The run will appear to be successful, but if you go to the logs there will be a warning that it failed as seen below:

2024-10-07 09:57:34 -0700   31813 execution          WARNING  Failed to execute aggregation function with error: {'message': "Execution failure in '__aggregate__': (Exception) Test", 'messageFormat': "Execution failure in '{func_name}': {error_type_and_message}", 'messageParameters': {'func_name': '__aggregate__', 'error_type_and_message': '(Exception) Test'}, 'referenceCode': 'Unknown', 'code': 'UserError', 'innerError': {'code': 'ScriptExecutionError', 'innerError': None}, 'debugInfo': {'type': 'ScriptExecutionError', 'message': "Execution failure in '__aggregate__': (Exception) Test", 'stackTrace': 'Traceback (most recent call last):\n', 'innerException': None}}
2024-10-07 09:57:34 -0700   31813 execution          WARNING  The flow will have empty metrics.

Expected behavior I would expect a warning to be raised to the user's attention on the cli. The prompt flow should still write the outputs from the run.

Running Information(please complete the following information): { "promptflow": "1.15.0", "promptflow-azure": "1.15.0", "promptflow-core": "1.15.0", "promptflow-devkit": "1.15.0", "promptflow-tracing": "1.15.0" }

Python (Darwin) 3.11.9

github-actions[bot] commented 1 week ago

Hi, we're sending this friendly reminder because we haven't heard back from you in 30 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 7 days of this comment, the issue will be automatically closed. Thank you!

jomalsan commented 5 days ago

bump