search

Arize-ai / phoenix

AI Observability & Evaluation
https://docs.arize.com/phoenix
Other
3k stars 216 forks source link

🗺️ Human / Programatic Feedback #2513

Open mikeldking opened 3 months ago

mikeldking commented 3 months ago

Add the ability to add human feedback via an API call

In many applications, but even more so for LLM applications, it is important to collect user feedback to understand how your application is performing in real-world scenarios. The ability to observe user feedback along with trace data can be very powerful to drill down into the most interesting datapoints, then send those datapoints for further review, automatic evaluation, or even datasets.

Phoenix should make it easy to attach user feedback to traces. It's often helpful to expose a simple mechanism (such as a thumbs-up, thumbs-down button) to collect user feedback for your application responses. The phoenix SDK or API should support sending feedback.

client

Openinference

REST

GraphQL

UI

dosubot[bot] commented 3 months ago

It seems like you've got this under control, if you want help or have specific questions, let me know what I can do for you!

#### About Dosu This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot.
stdweird commented 3 months ago

@mikeldking besides human thumbs up/down, there is also the other cases like "normal" ragas usage, where you (re)run same set of questions with grounds thruths as a baseline, and then want to add the evaluation scores to the trace.

the main issue is that there is no way that eg openinference callback communicates the span_id or trace_id back to the framework client in anyway (in our case llama-index). it guess the framework should provide a hook where eg the callback can set the id, and then somehow client code can retrieve it. i can guess the id based on the response text and the timing, but that ivolves getting dataframes back from phoenix, inspecting them with sole purpose to find the correct id.

once the id is found, afaik, i only have to construct a single row evaluation dataframe and then do a log_evaluations

mikeldking commented 3 months ago

@stdweird yes exactly. We can definitely make an ID be available in the application so that subsequent feedback and evaluations can be programmatically logged to the phoenix server.

From a roadmap perspective we do need to tackle two key things - #2340 and persistence so that we can make phoenix scale forward. We will be unblocked to work on this after.

Thanks for your insight. appreciate it!