[Q] How to use W&B prompts with Langchain chains?

[makeshn]

Hello,

How do we use W&BTracer with Langchain chains (not agents)?

When I try to use it like

from langchain import PromptTemplate, OpenAI, LLMChain

prompt_template = "What is a good name for a company that makes {product}?"

llm = OpenAI(temperature=0)
tracer = WandbTracer(wandb_config)

llm_chain = LLMChain(
    llm=llm,
    prompt=PromptTemplate.from_template(prompt_template),
    callbacks=[tracer]
)
llm_chain("colorful socks")

I get a warning that says

wandb: WARNING Skipping trace saving - unable to safely convert LangChain Run into W&B Trace due to: 'NoneType' object has no attribute 'lower'

and nothing gets logged to the wandb run. Could you please share an example of how to use W&BTracer with a Langchain chain?

langchain-0.0.209 wandb version - 0.15.4

code was run in the Quickstart Colab notebook. (https://colab.research.google.com/github/wandb/examples/blob/master/colabs/prompts/WandB_Prompts_Quickstart.ipynb)

[ArtsiomWB]

Hi @makeshn! What are you running your script on? Is it A notebook, or a regular IDE? As well as are you running this on your local, self-hosted instance, or on wandb public cloud?

[makeshn]

I am running it on Colab using the Quickstart Colab notebook. (https://colab.research.google.com/github/wandb/examples/blob/master/colabs/prompts/WandB_Prompts_Quickstart.ipynb)

[morganmcg1]

Hi @makeshn! Morgan from W&B here. We recently upgraded the Prompts integration.

For langchain, this is now the preferred way to log to W&B

os.environ["LANGCHAIN_WANDB_TRACING"] = "true"

https://python.langchain.com/docs/ecosystem/integrations/agent_with_wandb_tracing

Please let me know if this isn't working for you :)

[jlewi]

This worked for me. Is there away to do this with code/context managers without messing with the environment?

[morganmcg1]

Sure! The docs above also show a context manager:

‘’’ from langchain.callbacks import wandb_tracing_enabled

with wandb_tracing_enabled(): agent.run("What is 5 raised to .123243 power?") # this should be traced ‘’’

[trivikramak]

The docs still show an example of an Agent use case. Does it work with a simple LLMChain? (NOT Agents)

EDIT: It's working. Just checked with a SequentialChain. Thanks!

[ArtsiomWB]

Awesome! Would you like us to close this ticket for you?

[cvillafraz]

How can I log custom metrics when a chain runs using the wandb tracer or the context manager?

[morganmcg1]

@cvillafraz you can pass additional metrics like so:

# start a wandb run (this is usually triggered by langchain, but we need to do it here first)
import wandb
wandb.init(project="my-project-name")

wandb.log({"my-custom-metric": 0.88}, commit=False)
agent.run("do a thing")

This will give you an additional column in the Traces tables with your custom metric.

Whats happening is that with commit=False, wandb takes note of what you want to log, but doesn't actually send it to the server until the call to your agent is made and it's inputs and outputs are logged

[cvillafraz]

@cvillafraz you can pass additional metrics like so:

# start a wandb run (this is usually triggered by langchain, but we need to do it here first)
import wandb
wandb.init(project="my-project-name")

wandb.log({"my-custom-metric": 0.88}, commit=False)
agent.run("do a thing")

This will give you an additional column in the Traces tables with your custom metric.

Whats happening is that with commit=False, wandb takes note of what you want to log, but doesn't actually send it to the server until the call to your agent is made and it's inputs and outputs are logged

Thank you! Now, how can I log a metric calculated with the agent output in the same row? Let's say my agent outputs a list of values, and I want to log the mean of the values in the same row as the agent output.

[ayulockin]

Hey @makeshn, W&B Trace is an old product offering which is not maintained anymore. We were focused on shipping a better tooling called W&B Weave. Team is working on the Langchain integration.

Weave announcement Weave Documentation

Closing this issue for now. Do let me know if you need anything more.