LLM provider traces (e.g. OpenAI) don't handle images in the messages array

[willbakst]

Description

You can include images for vision with certain LLM provider APIs like OpenAI. The structure is slightly different and as a result it's not showing up in the UI. This also means that content sent in this structure won't show up even if it's just text.

Here is an example of the code that I would like to render images but instead only shows the raw json:

messages = [
    {"role": "user", "content": [{"type": "text", "text": "Recommend a book"}]
    {"role": "user", "content": [{"type": "image_url", {"url": "https://..."}}]
]

Neither of the above show up pretty in the UI like I would like. Taking a look at the code, it seems like this is because the UI only handles the non-list str messages array structure e.g. {"role": "user", "content": "Recommend a book"}

Python, Logfire & OS Versions, related packages (not required)

No response

[PauloFavero]

I have a similar issue when using the assistants and thread api's.

Log:Unable to instrument OpenAI API call: Expectedoptions.json_datato be a dictionary

[dmontagu]

@PauloFavero I think that is a separate issue from the one @willbakst is mentioning in this issue, which is a pure frontend problem. If you could provide some additional context about how to reproduce the issue we can look into it. It probably makes sense to create a separate GitHub issue for that though.

@willbakst I believe I have a fix for this for the "default" panel, but I'd like to coordinate with you (or Brendan or anyone else from Mirascope) how to modify the Mirascope version of that panel to achieve the same thing.

I know I still owe you guys a way to submit updates via a public repository; I am hoping to have a significant chuck of time to invest in custom panels infrastructure early next week.

[willbakst]

@dmontagu I appreciate that. Let me know how you want us to work with you here to update our panels.