search

microsoft / promptflow

Build high-quality LLM apps - from prototyping, testing to production deployment and monitoring.
https://microsoft.github.io/promptflow/
MIT License
9.14k stars 829 forks source link

Deployed Endpoint App Insights Not Capturing Full Data [BUG] #3563

Open adampolak-vertex opened 1 month ago

adampolak-vertex commented 1 month ago

Describe the bug The application insights for PromptFlow as a deployed endpoint cuts off the full data.

When the content provided to the prompt is too long it cuts it off and does not provide the full data payload.

How To Reproduce the bug Steps to reproduce the behavior, how frequent can you experience the bug:

  1. Deploy a PromptFlow endpoint
  2. Add an input argument which requires 24611 characters (or a large number)

Expected behavior Would expect all the content used in the PromptFlow would be included in application insights, but these fields are cut off.

Additional context I understand that there may be size/cost issues with retaining all data. But there should be some kind of notice like TRUNCATED DUE TO SIZE so it does not appear that there was an error in processing.

brynn-code commented 1 month ago

Hi @adampolak-vertex , thanks for reaching us. Promptflow will not limit data size, I guess the data size was limited by Azure App Insights. I found some reference about this limitation: https://learn.microsoft.com/en-us/answers/questions/543396/how-to-increase-the-size-of-messages-logged-in-app https://learn.microsoft.com/en-us/azure/azure-monitor/app/api-custom-events-metrics#limits

We will update this limitation to our docs. + MS learn deployment doc owner: @likebupt

github-actions[bot] commented 2 weeks ago

Hi, we're sending this friendly reminder because we haven't heard back from you in 30 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 7 days of this comment, the issue will be automatically closed. Thank you!