Error when using ToolChoiceOption REQUIRED in ThreadRunRequest builder with Assistants API

[zhengzheli]

Hi,

Thank you for making this library for Java. I was trying to require my Assistant to use file search every time. However, I got an error. Am not sure if it's because I'm using this code incorrectly.

  Stream<Event> runStream = BaseConfig.simpleOpenAI.threadRuns()
                    .createStream(threadId, ThreadRunRequest.builder()
                            .assistantId(CommonConsts.MEDICAL_PHARMACY_ASSISTANT_ID)
                            .toolChoice(ToolChoiceOption.REQUIRED)
                            .tool(AssistantTool.fileSearch(10))
                            .build())
                    .join();

Here is what the request looks like. Sorry for poor formatting:

additionalInstructions: null
additionalMessages: Collections$EmptyList@21658 size=0
assistantId: "xxx"
instructions: null
maxCompletionTokens: null
maxPromptTokens: null
metadata: null
model: null
parallelToolCalls: null
responseFormat: null
stream: null
temperature: null
toolChoice: ToolChoiceOption@21639 "REQUIRED"
tools: Collections$SingletonList@23521 size=1
0: AssistantTool@23425 "AssistantTool(fileSearch=null)"
fileSearch: null
function: null
type: ToolType@23527 "FILE_SEARCH"
topP: null
truncationStrategy: null
arguments[1]
ThreadRunRequest@23520
additionalInstructions: null
additionalMessages: Collections$EmptyList@21658 size=0
assistantId: "xxx"
instructions: null
maxCompletionTokens: null
maxPromptTokens: null
metadata: null
model: null
parallelToolCalls: null
responseFormat: null
stream: null
temperature: null
toolChoice: ToolChoiceOption@21639 "REQUIRED"
tools: Collections$SingletonList@23521 size=1
topP: null
truncationStrategy: null

For some reason, when I get the response with ThreadRunStepDelta, it says:

Exception in thread "pool-11-thread-1" io.github.sashirestela.cleverclient.support.CleverClientException: Cannot convert the Json

{
    "id": "step_6A9b9pXkKuTRx03L9jp4bqXq",
    "object": "thread.run.step.delta",
    "delta": {
        "step_details": {
            "type": "tool_calls",
            "tool_calls": [
                {
                    "index": 0,
                    "id": "call_3kPyOYj5e7CC83Opf9zZh4D2",
                    "type": "browser",
                    "browser": {}
                }
            ]
        }
    }
}

to class io.github.sashirestela.openai.domain.assistant.ThreadRunStepDelta.

Strangely enough the tool call type is "browser". This is not a valid type.

Caused by: com.fasterxml.jackson.databind.exc.InvalidTypeIdException: Could not resolve type id 'browser' as a subtype of io.github.sashirestela.openai.domain.assistant.StepDetail$ToolCallsStep$StepToolCall: known type ids = [code_interpreter, file_search, function] (for POJO property 'tool_calls') at [Source: (String)"{"id":"step_RNUV1jww67EXCHDN3KgjrypM","object":"thread.run.step.delta","delta":{"step_details":{"type":"tool_calls","tool_calls":[{"index":0,"id":"call_sUIS9vdT0yRiZ5mm42gwcIAG","type":"browser","browser":{}}]}}}"; line: 1, column: 186] (through reference chain: io.github.sashirestela.openai.domain.assistant.ThreadRunStepDelta["delta"]->io.github.sashirestela.openai.domain.assistant.ThreadRunStepDelta$RunStepDeltaDetail["step_details"]->io.github.sashirestela.openai.domain.assistant.StepDetail$ToolCallsStep["tool_calls"]->java.util.ArrayList[0])

[zhengzheli]

This is what the assistant it's querying looks like

[zhengzheli]

Oh, it seems to be working now that I added the Attachment to the ThreadMessageRequest.

    ```
BaseConfig.simpleOpenAI.threadMessages()
        .create(threadId, ThreadMessageRequest.builder()
                .role(ThreadMessageRole.USER).attachment(Attachment.builder()
                    .fileId(fileid)
                    .tool(AttachmentTool.FILE_SEARCH)
                    .build())
                .content(text)
                .build())
        .join();

[sashirestela]

@zhengzheli is awesome you could identify the problem by yourself. Thanks for using simple-openai.

[zhengzheli]

Sorry @sashirestela the issue seems to have returned. And it happens even when I just run the ConversationV2Demo sample code you provided for Assistants V2.

Exception in thread "main" io.github.sashirestela.cleverclient.support.CleverClientException: Cannot convert the Json (see below) to class io.github.sashirestela.openai.domain.assistant.ThreadRunStepDelta.

{
    "id": "step_m757CaIaVKBQyAJAaVGF4TJw",
    "object": "thread.run.step.delta",
    "delta": {
        "step_details": {
            "type": "tool_calls",
            "tool_calls": [
                {
                    "index": 0,
                    "id": "call_pzRHgJWwD5JD0SJ5icaYu7rZ",
                    "type": "file_search",
                    "file_search": {
                        "ranking_options": {
                            "ranker": "default_2024_08_21",
                            "score_threshold": 0.0
                        },
                        "results": []
                    }
                }
            ]
        }
    }
}

Caused by: com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize value of type java.lang.String from Object value (token JsonToken.START_OBJECT) at [Source: (String)"{"id":"step_m757CaIaVKBQyAJAaVGF4TJw","object":"thread.run.step.delta","delta":{"step_details":{"type":"tool_calls","tool_calls":[{"index":0,"id":"call_pzRHgJWwD5JD0SJ5icaYu7rZ","type":"file_search","file_search":{"ranking_options":{"ranker":"default_2024_08_21","score_threshold":0.0},"results":[]}}]}}}"; line: 1, column: 233] (through reference chain: io.github.sashirestela.openai.domain.assistant.ThreadRunStepDelta["delta"]->io.github.sashirestela.openai.domain.assistant.ThreadRunStepDelta$RunStepDeltaDetail["step_details"]->io.github.sashirestela.openai.domain.assistant.StepDetail$ToolCallsStep["tool_calls"]->java.util.ArrayList[0]->io.github.sashirestela.openai.domain.assistant.StepDetail$ToolCallsStep$StepToolCall$FileSearchToolCall["file_search"]->java.util.LinkedHashMap["ranking_options"])

Could you please check it?

[sunker0115]

I have encountered this situation as well. Exception in thread "demo-pool-84" io.github.sashirestela.cleverclient.support.CleverClientException: Cannot convert the Json to class io.github.sashirestela.openai.domain.assistant.ThreadRunStepDelta.

{
    "id": "step_UP2KebrVDWaWYqjZTs9lniY6",
    "object": "thread.run.step.delta",
    "delta": {
        "step_details": {
            "type": "tool_calls",
            "tool_calls": [
                {
                    "index": 0,
                    "id": "call_pu1KBQUR0ihL1uQCAukquAPR",
                    "type": "file_search",
                    "file_search": {
                        "ranking_options": {
                            "ranker": "default_2024_08_21",
                            "score_threshold": 0.0
                        },
                        "results": []
                    }
                }
            ]
        }
    }
}

Caused by: com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize value of type java.lang.String from Object value (token JsonToken.START_OBJECT) at [Source: REDACTED (StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION disabled); line: 1, column: 233] (through reference chain: io.github.sashirestela.openai.domain.assistant.ThreadRunStepDelta["delta"]->io.github.sashirestela.openai.domain.assistant.ThreadRunStepDelta$RunStepDeltaDetail["step_details"]->io.github.sashirestela.openai.domain.assistant.StepDetail$ToolCallsStep["tool_calls"]->java.util.ArrayList[0]->io.github.sashirestela.openai.domain.assistant.StepDetail$ToolCallsStep$StepToolCall$FileSearchToolCall["file_search"]->java.util.LinkedHashMap["ranking_options"]) at com.fasterxml.jackson.databind.exc.MismatchedInputException.from(MismatchedInputException.java:59) at com.fasterxml.jackson.databind.DeserializationContext.reportInputMismatch(DeserializationContext.java:1767) at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1541) at com.fasterxml.jackson.databind.DeserializationContext.handleUnexpectedToken(DeserializationContext.java:1446) at com.fasterxml.jackson.databind.DeserializationContext.extractScalarFromObject(DeserializationContext.java:958) at com.fasterxml.jackson.databind.deser.std.StdDeserializer._parseString(StdDeserializer.java:1424) at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:48) at com.fasterxml.jackson.databind.deser.std.StringDeserializer.deserialize(StringDeserializer.java:11) at com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringKeyMap(MapDeserializer.java:623) at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:449) at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:32) at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:138) com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:310) at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:215) at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:187) at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:170) at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:136) at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:263) at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer._deserializeFromArray(CollectionDeserializer.java:361) at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:244) at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:28) at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:138) at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:310) at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:215) at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:187) at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:170) at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:136) at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeWithType(BeanDeserializerBase.java:1380) at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:147) at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:310) at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177) at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:138) at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:310) at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177) at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:342) at com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:2125) at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1566) at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1586) at io.github.sashirestela.cleverclient.util.JsonUtil.jsonToObject(JsonUtil.java:46)

[sashirestela]

Hi @zhengzheli and @sunker0115 Let me take a look at this issue which could be related to new functionality delivered by OpenAI yesterday.

[zhengzheli]

Thanks for the quick fix @sashirestela !