[ENHANCEMENT] User awareness in traces

[stdweird]

We want to give users access to the traces they generate via a webapplication that uses llama-index.

We currently see 2 issues: a. can we inject certain data in the metadata of the spans? we were thinking eg the username. this would give us also a way to filter all spans/traces of a single user b. can we get the tracedetail with only the tracedetail? we would give the users a url like http://.../projects/UHJvabcjdDoy/traces/ebf94c5d736a5e72a47c680b7edefgb6, and have broad authentication on the the ./projects/../traces url, assuming that only the user who can guess the traceid (or gets it from our app ;) can see it. right now, this tracedetail url also load the main projects page, and is like a popup rather then a standalone page.

if a can be implemented, an url that would filter spans based on metadata would also be good, so we can filter the spans that the user generated themself.

[dosubot[bot]]

Hey @stdweird! Great to see you back here - hope you're doing awesome. 😄 Ready to dive into another tech adventure together?

Based on the current structure and capabilities of the Arize Phoenix repository, it is indeed possible to implement the enhancements you've suggested. Here's a breakdown of how each of your requests could be addressed:

1. Injecting User Data into Span Metadata: While the current codebase doesn't explicitly show how to inject metadata into spans, the concept of adding metadata (such as a username) to spans is a common practice in distributed tracing. This usually involves modifying the span creation or span processing logic to include additional metadata. Since the repository already deals with spans and their attributes, it's feasible to extend this functionality to include user-specific metadata.

2. Accessing and Filtering User-Generated Traces: The SpansTableSpansQuery$variables type definition includes filterCondition, which can be used to filter traces based on specific criteria. This indicates that the backend supports filtering, which can be exposed through a web application interface allowing users to specify their filtering criteria.

3. Standalone Trace Detail Page: The structure of the query and the use of FragmentRefs suggest that the system is designed with modularity in mind, using GraphQL. This design can facilitate the development of a standalone trace detail page. By creating a new query or modifying existing ones to fetch all necessary details for a single trace, a standalone page can be implemented without loading the main project's page. This approach would leverage the existing infrastructure (like the SpansTableSpansQuery) to fetch detailed information for an individual trace.

Regarding the URL that would filter spans based on metadata, this would be a natural extension of the filtering capabilities mentioned above. Once the user-specific metadata is injected into the spans, a filter condition could be created to filter spans based on this metadata.

Please note that while these enhancements are technically feasible, they would require significant changes to the existing codebase and thorough testing to ensure they don't introduce any regressions or new issues.

For more details, you can refer to the following files in the repository:

• app/src/pages/project/generated/SpansTableSpansQuery.graphql.ts
• src/phoenix/utilities/span_store.py

I hope this provides a clear path forward for your enhancement requests. If you have any further questions or need additional clarification, please don't hesitate to ask.

Sources

• app/src/pages/project/generated/SpansTableSpansQuery.graphql.ts
• src/phoenix/utilities/span_store.py

#### About Dosu

This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot.

[axiomofjoy]

Hey @stdweird, thanks for the message.

a. can we inject certain data in the metadata of the spans? we were thinking eg the username. this would give us also a way to filter all spans/traces of a single user

As far as I know, LlamaIndex currently does not allow users to supply metadata as a field on their query engine invocations. We've mentioned this to the LlamaIndex team and can bring it up again.

b. can we get the tracedetail with only the tracedetail? we would give the users a url like http://.../projects/UHJvabcjdDoy/traces/ebf94c5d736a5e72a47c680b7edefgb6, and have broad authentication on the the ./projects/../traces url, assuming that only the user who can guess the traceid (or gets it from our app ;) can see it. right now, this tracedetail url also load the main projects page, and is like a popup rather then a standalone page.

It sounds like you want the trace details to be their own page rather than a pullover so that you can display the page to your users? Can you clarify what you have in mind in terms of authentication?

if a can be implemented, an url that would filter spans based on metadata would also be good, so we can filter the spans that the user generated themself.

We have support for metadata and filters in Phoenix. The blocker in this case is in getting the metadata associated with traces produced by LlamaIndex.

[stdweird]

hey @axiomofjoy thanks for the feedback

• wrt the metdata: where would this data be injected ideally? i already have a custom simplespanprocessor, where i do something with all spans that pass. is that a good place inject the metadata? (until llama-index provides somethng better built in ofcourse)
• wrt the authentication: that depends what we can work with. i would do the auth handling in an apache proxy, so eg mapping http://myhost/traces/user/abcdef to http://myhost/traces/user//abcdef. so phoenix could trust the url to filter on eg user. i could also have apache inject some metadata that phoenix can pick up from the header, and eg http://myhost/traces/user would only respond with the user data from that header value. (but i am no expert, so besides some simple mechanisms i can't propose anything).

[axiomofjoy]

Thanks for the reply @stdweird.

wrt the metdata: where would this data be injected ideally? i already have a custom simplespanprocessor, where i do something with all spans that pass. is that a good place inject the metadata? (until llama-index provides somethng better built in ofcourse)

You can add metadata as a JSON object under the metadata attribute. See the semantic conventions here.

[axiomofjoy]

Can you help me understand from a product perspective how your end-users benefit from viewing traces? We typically think of the end-user of Phoenix as the AI application developer (yourself), not the end-user of the AI application (e.g., the person interacting with a chat interface). This sounds like a new use-case.

[stdweird]

we develop the app mainly for a support team who helps out the endusers. this support team are also the document maintainers and are pretty knowledgeable.

we want to give the support team people access to their own traces so they can help us spot issues with eg the parsing of the sources, or the construction of indices of pipeline of agents etc. so they can test and "debug" the response process.

it's unclear (GDPR wise) if we can give the support team access to all traces, and setting up duplicate infra is not what we want.

for the endusers it might be overkill or even confusing to see the traces, but otoh, we suspect that endusers will either not look at the traces or that some powerusers will also give us valuable feedback if things go wrong.

[axiomofjoy]

@stdweird Thanks for the clarification, makes sense. It sounds like what you might be after is incorporating RBAC. I'm unsure if RBAC is on the Phoenix roadmap, since we intend to keep the application as lightweight and easy to set up as possible. @mikeldking Can you provide some detail here?

[stdweird]

@axiomofjoy well ofcourse, phoenix with builtin RBAC is what we want ;)

but i understand it's not that (yet?), so something simpler is already more then welcome.

[mikeldking]

@stdweird thanks for your use-case. It's an interesting one - we don't have RBAC nor giving access to traces to specific users on our current roadmap and I think it might diverge a bit from our primary end-user as described by Xander so we can't really promise anything on this front.

One possible solution is for you to leverage Phoenix's projecting capabilities. This means that you could enforce restrictions at the load balancer level to segment user traffic by /projects/xyz where each project would correspond to a user. Not sure how easy it would be to implement this at the load balancer level but you certainly can try. Phoenix does use GraphQL so you could use it to resolve which project corresponds to which user.

I think otherwise your only solution right now is going to be to fork the project for RBAC.

User-awereness in general will be coming in some capacity and you can definitely add any attributes you may need via OpenTelemetry https://docs.arize.com/phoenix/tracing/how-to-tracing/custom-spans