Issue: Unable to mask secrets using distributed tracing

[bhanson-techempower]

Context

We're using Prompt Flow in our production application and are evaluating LangSmith for observability. Prompt Flow spawns worker threads to distribute work and we're trying to use LangSmith to get visibility in those threads.

We've been able to get threads (OS threads, not LangSmith Threads) working under a unified LangSmith Trace by using distributed tracing. However, LangSmith by default has a very aggressive serializer; it is converting in-memory Prompt Flow connection objects to JSON that were never intended to be serialized as they contain secrets.

We've been able to get the hide_inputs functionality working in order to mask this data, but cannot get it to work with distributed tracing. We suspect the distributed tracing functionality is short-circuiting somewhere before the custom hide_inputs function gets assigned.

Observed Behavior

Enabling hide_inputs works as expected when configured in the main and child threads:

Without changing anything else, adding a parent to set distributed tracing causes the threads to be grouped together appropriately under one Trace, but the hide_inputs functionality stops working:

Expected Behavior

Configuring both options would group them under the same Trace and also utilize the configured hide_inputs.

Steps to Reproduce

I've uploaded a repo with a minimal example that demonstrates the behavior we're seeing.

https://github.com/bhanson-techempower/langsmith-sdk-bug

Version Info

• python 3.11.3
• langsmith 0.1.71 (also observed on 0.1.59)

Other Comments

We're open to ideas on how to orchestrate LangSmith with Prompt Flow. We didn't find any information in the documentation on Prompt Flow integration nor multi-threading, and the distributed tracing section of the docs assumes a web framework. LangSmith looks like a great option, but integrating with Prompt Flow has not been simple and we've had to dig in the source a bit to make progress.

If this is a misconfiguration, then raising an exception when both options are present would be helpful.

Alternatively, if LangSmith shipped with default secrets masking capability that would be fantastic.

Thank you!

[hinthornw]

Thank you for the detailed repro code! Will address this asap

[bhanson-techempower]

Thank you for the lightning fast turnaround! Confirmed fixed. 🎉

[hinthornw]

awesome - thanks for the confirmation! Let us know how everything goes w/ the promptflow + langsmith integration!