Closed tiaan720 closed 5 days ago
vbarda
commented
5 days ago @tiaan720 i am not sure why you're seeing any randomness in the results. i tested your example with gpt-4o and it works as i would expect: the artifact is never actually sent to the LLM but the tool message with artifact IS streamed from the graph
artifacts are handled internally by langchain and the LLM never sees them, so you can't control them via the prompts
tiaan720
commented
5 days ago Good day @vbarda, thanks for the reply. This issue is that the output artifact from the tool should not be regurgitated back to the model in the final response. With randomness I implied that the bug doesn't always occur when running the code. We know that LLM's inherently have a randomness aspect to it, even thought temperature is set to zero. So maybe give it a couple of tries and see if you can get the bug to arise (remember to use gpt-4 as I mentioned in my previous response). If you cant get it to work I will try and capture screenshots or send a video.
vbarda
commented
5 days ago ah @tiaan720 i actually see the issue even with gpt-4o -- i think the model is just hallucinating the image data, so you would need to improve the prompt / tool descriptions. try replacing the return from circle_plot with fake image data
@tool(response_format="content_and_artifact")
def circle_plot(title: str, radius: float) -> Tuple[str, str]:
...
return title, "foo"i get this in response from gpt-4 when using a fake artifact.
{'agent': {'messages': [AIMessage(content='The area of the circle with a radius of 5 is 78.54. Here is a plot of the circle:\n\n', ...}
so i don't believe the artifact is actually being fed into the model or that there is any other bug in langgraph. will close for now but let me know if there is still an issue
Checked other resources
Example Code
Error Message and Stack Trace (if applicable)
Description
We experienced in our code that the artifact output of the tool is streamed back into the end response. According to the documentation this should not be the case:
param artifact: Any = None# Artifact of the Tool execution which is not meant to be sent to the model. Should only be specified if it is different from the message content, e.g. if only a subset of the full tool output is being passed as message content but the full output is needed in other parts of the code.
We created an example python script to demonstrate this. This bug usually happens at random and for you to notice the issue you will have to try and rerun it a couple of times. If it doesnt work try setting it to gpt-4o, then run it and then set it back to gpt-4 and run it again. The issue didnt really occur with gpt-4o and its an edge case but it can cause alot of damage if it occurs. Obviously we can resolve it by simply using a different model and changing the prompt or prompting the artifact not to be streamed back. The core issue is still present that the artifact should never be fed back to the model. The tokens streamed back can occur high costs in a production environment. We also conducted mlflow traces with screenshots attached to demonstrate what is happening. We recommend running the code and one will see in the terminal what is happening as well.
System Info
mlflow==2.18.0 numpy==1.26.4 plotly==5.24.1 langchain-openai==0.2.1 langgraph==0.2.35 langchain==0.3.7