search

microsoft / promptflow

Build high-quality LLM apps - from prototyping, testing to production deployment and monitoring.
https://microsoft.github.io/promptflow/
MIT License
9.47k stars 866 forks source link

[Feature Request] Improve Evaluation Charts #3774

Closed hayescode closed 1 week ago

hayescode commented 1 month ago

Is your feature request related to a problem? Please describe. My evaluation flow has inputs like "source system" and "difficulty" that i'd like charts for to visually see progress across evaluations. The only fields that are available in the Custom Charts however are evaluation flow outputs, not even my custom metrics.

Describe the solution you'd like Any field in the evaluation flow results should be available for custom charts, including custom metrics (aggregations).

Additional context image image

vinuthakaranth commented 1 month ago

Hi @hayescode , thanks for your valuable feedback in enhancing our product.

We are trying to understand what 'including custom metrics (aggregations)' means here. How are you envisioning having this in chart? Any further details on this would definitely help in putting this request to implementation.

Could you give us an example of x-Axis and y-axis combinations you would like to see in your custom chart and how that would help you in viewing progress?

hayescode commented 1 month ago

Hi @vinuthakaranth - I have dimensions like difficulty (easy/medium/hard) I'd like to so the accuracy across for example. Another example would be RAG source, so I can see source A has high accuracy but not source B, soay e I should look into Source B instead of anything with the prompting/LLM.

github-actions[bot] commented 2 weeks ago

Hi, we're sending this friendly reminder because we haven't heard back from you in 30 days. We need more information about this issue to help address it. Please be sure to give us your input. If we don't hear back from you within 7 days of this comment, the issue will be automatically closed. Thank you!