Arize-ai / phoenix

AI Observability & Evaluation
https://docs.arize.com/phoenix
Other
3.83k stars 284 forks source link

[Playground][BUG] playground UI shows `ZodError` for openai span with `response_format` #5247

Open RogerHYang opened 1 day ago

RogerHYang commented 1 day ago

playground UI shows error for the following span

from openai import OpenAI
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor

from openinference.instrumentation.openai import OpenAIInstrumentor

endpoint = "http://127.0.0.1:4317"
tracer_provider = TracerProvider()
tracer_provider.add_span_processor(SimpleSpanProcessor(OTLPSpanExporter(endpoint)))
OpenAIInstrumentor().instrument(tracer_provider=tracer_provider)

client = OpenAI()
MODEL = "gpt-4o-mini"

math_tutor_prompt = """
You are a helpful math tutor. You will be provided with a math problem,
and your goal will be to output a step by step solution, along with a final answer.
For each step, just provide the output as an equation use the explanation field
to detail the reasoning.
"""
question = "how can I solve 8x + 7 = -23"
messages = [
    {"role": "system", "content": math_tutor_prompt},
    {"role": "user", "content": question},
]

response_format = {
    "type": "json_schema",
    "json_schema": {
        "name": "math_reasoning",
        "schema": {
            "type": "object",
            "properties": {
                "steps": {
                    "type": "array",
                    "items": {
                        "type": "object",
                        "properties": {
                            "explanation": {"type": "string"},
                            "output": {"type": "string"},
                        },
                        "required": ["explanation", "output"],
                        "additionalProperties": False,
                    },
                },
                "final_answer": {"type": "string"},
            },
            "required": ["steps", "final_answer"],
            "additionalProperties": False,
        },
        "strict": True,
    },
}

client.chat.completions.create(
    model=MODEL,
    messages=messages,
    response_format=response_format,
)
Parker-Stafford commented 16 hours ago

what error does this show?

RogerHYang commented 15 hours ago

here's the error

Screenshot 2024-11-01 at 10 09 29 AM
cephalization commented 14 hours ago

Ah, this is because our frontend invocation param schema is too restrictive.

/**
 * Model generic invocation parameters schema in zod.
 */
const invocationParameterSchema = z.record(
  z.union([z.boolean(), z.number(), z.string(), z.array(z.string())])
);

This needs to include z.record(z.unknown()) in the union

https://github.com/Arize-ai/phoenix/blob/2fd29480409206fe2483ac2ba7dd41949e68fd2e/app/src/pages/playground/schemas.ts#L106-L111