OllamaStreamingChatLanguageModel should support tools

[edeandrea]

Currently the OllamaStreamingChatLanguageModel doesn't support tool integration. Let's add it.

I'll take this and work on it.

[geoand]

Tools are not supported in any streaming chat.

It's like that in upstream LangChain4j too as well, and honestly I don't see how they could be supported... But if you can find a clever way, that would be great!

[edeandrea]

Thats not true. OpenAI supports it as do a bunch of others.

[geoand]

I seriously doubt it, but I'd love to be proven wrong

[edeandrea]

https://github.com/langchain4j/langchain4j/blob/118b15423c883c56c533cd9d2ee366382d1700e6/langchain4j-open-ai/src/main/java/dev/langchain4j/model/openai/OpenAiStreamingChatModel.java#L135-L243

[geoand]

Yeah, I was pretty sure that's what you were looking at.

That is going to buy you nothing when using the AI service - which is what I was referring to

[geoand]

So you can certainly go ahead and add the tool support to the streaming chat model in the Ollama case, but don't expect it to have any effect in your application (which behaves the same way as if you used OpenAI)

[edeandrea]

That is going to buy you nothing when using the AI service - which is what I was referring to

I guess I don't completely understand. Can you explain a bit more?

[edeandrea]

is it because of the back and forth thats needed in the case of tools? i.e. client makes call, provider responds with something saying "call this tool". client invokes tool, calls back to provider, etc, until provider comes back with complete response?

[geoand]

Precisely!

[edeandrea]

So then going back to https://github.com/langchain4j/langchain4j/blob/118b15423c883c56c533cd9d2ee366382d1700e6/langchain4j-open-ai/src/main/java/dev/langchain4j/model/openai/OpenAiStreamingChatModel.java#L135-L243, how does that work and solve that use case?

[geoand]

That only works for a specific API call - that's what that abstraction does. But in order to use tools, you need the entire dance that is done in the AI service implementation

[edeandrea]

That makes sense. I actually just tried it with OpenAI...

2024-07-30 14:58:14,641 INFO  [io.qua.lan.ope.OpenAiRestApi$OpenAiClientLogger] (vert.x-eventloop-thread-1) Request:
- method: POST
- url: https://parasol-chat-aiworkshop.apps.cluster-wk5tx.sandbox3051.opentlc.com/v1/chat/completions
- headers: [Accept: text/event-stream], [Authorization: Be...my], [Content-Type: application/json], [User-Agent: langchain4j-openai], [content-length: 2538]
- body: {
  "model" : "parasol-chat",
  "messages" : [ {
    "role" : "system",
    "content" : "You are a helpful, respectful and honest assistant named \"Parasol Assistant\".\nYou will be given a claim summary, references to provide you with information, and a question. You must answer the question based as much as possible on this claim with the help of the references.\nAlways answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.\n\nIf a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.\n"
  }, {
    "role" : "user",
    "content" : "    Claim ID: 1\n\n    Claim Summary:\n    \n    On January 2nd, 1955, at around 3:30 PM, a car accident occurred at the intersection of Colima Road and Azusa Avenue in Hill Vallet. The involved parties were Marty McFly, driving a silver Delorean DMC-12 (OUTA-TIME), and Biff Tanner in a blue Type 2 Volkswagen Bus (BIF-RULZ).\n\n    Marty was heading south on Colima Road when Biff failed to stop at the red traffic signal on Asuza Avenue, causing a collision with Marty's vehicle. Both drivers exchanged information and took photos of the accident scene, which included damages to the front driver and passenger side of Marty's Delorean DMC-12 and the front driver's side of Biff's Volkswagen Bus. No injuries were reported.\n\n    Marty has attached necessary documents, such as photos, a police report, and an estimate for repair costs, to his email. He requests prompt attention to the claim and is available at (916) 555-4385 or marty.mcfly@email.com for any additional information or documentation needed.\n    \n\n    Question: Should I approve this claim?\n"
  } ],
  "temperature" : 0.3,
  "top_p" : 1.0,
  "stream" : true,
  "stop" : [ "DONE", "done", "stop", "STOP" ],
  "presence_penalty" : 0.0,
  "frequency_penalty" : 0.0,
  "tools" : [ {
    "type" : "function",
    "function" : {
      "name" : "updateClaimStatus",
      "description" : "update claim status",
      "parameters" : {
        "type" : "object",
        "properties" : {
          "claimId" : {
            "type" : "integer"
          },
          "status" : {
            "type" : "string"
          }
        },
        "required" : [ "claimId", "status" ]
      }
    }
  } ]
}

2024-07-30 14:58:14,702 ERROR [io.qua.mut.run.MutinyInfrastructure] (vert.x-eventloop-thread-1) Mutiny had to drop the following exception: io.smallrye.mutiny.CompositeException: Multiple exceptions caught:
        [Exception 0] dev.ai4j.openai4j.OpenAiHttpException: {"object":"error","message":"[{'type': 'extra_forbidden', 'loc': ('body', 'tools'), 'msg': 'Extra inputs are not permitted', 'input': [{'type': 'function', 'function': {'name': 'updateClaimStatus', 'description': 'update claim status', 'parameters': {'type': 'object', 'properties': {'claimId': {'type': 'integer'}, 'status': {'type': 'string'}}, 'required': ['claimId', 'status']}}}]}]","type":"BadRequestError","param":null,"code":400}
        [Exception 1] java.lang.NullPointerException: Cannot invoke "dev.langchain4j.model.output.Response.content()" because "response" is null
        at io.smallrye.mutiny.subscription.Subscribers$CallbackBasedSubscriber.onFailure(Subscribers.java:95)
        at io.smallrye.mutiny.operators.multi.builders.BaseMultiEmitter.failed(BaseMultiEmitter.java:89)
        at io.smallrye.mutiny.operators.multi.builders.BufferItemMultiEmitter.drain(BufferItemMultiEmitter.java:108)
        at io.smallrye.mutiny.operators.multi.builders.BufferItemMultiEmitter.failed(BufferItemMultiEmitter.java:62)
        at io.smallrye.mutiny.operators.multi.builders.BaseMultiEmitter.fail(BaseMultiEmitter.java:78)
        at io.smallrye.mutiny.operators.multi.builders.SerializedMultiEmitter.drainLoop(SerializedMultiEmitter.java:110)
        at io.smallrye.mutiny.operators.multi.builders.SerializedMultiEmitter.drain(SerializedMultiEmitter.java:92)
        at io.smallrye.mutiny.operators.multi.builders.SerializedMultiEmitter.onFailure(SerializedMultiEmitter.java:77)
        at io.smallrye.mutiny.operators.multi.builders.SerializedMultiEmitter.fail(SerializedMultiEmitter.java:149)
        at org.jboss.resteasy.reactive.client.impl.MultiInvoker.lambda$method$0(MultiInvoker.java:127)
        at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
        at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
        at org.jboss.resteasy.reactive.client.impl.RestClientRequestContext.handleUnrecoverableError(RestClientRequestContext.java:368)
        at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.handleException(AbstractResteasyReactiveContext.java:329)
        at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:175)
        at org.jboss.resteasy.reactive.client.impl.RestClientRequestContext$1.lambda$execute$0(RestClientRequestContext.java:314)
        at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:279)
        at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:261)
        at io.vertx.core.impl.ContextInternal.lambda$runOnContext$0(ContextInternal.java:59)
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:1583)
        Suppressed: java.lang.NullPointerException: Cannot invoke "dev.langchain4j.model.output.Response.content()" because "response" is null
                at dev.langchain4j.model.openai.InternalOpenAiHelper.removeTokenUsage(InternalOpenAiHelper.java:283)
                at dev.langchain4j.model.openai.OpenAiStreamingChatModel.createResponse(OpenAiStreamingChatModel.java:251)
                at dev.langchain4j.model.openai.OpenAiStreamingChatModel.lambda$generate$5(OpenAiStreamingChatModel.java:217)
                at io.smallrye.mutiny.subscription.Subscribers$CallbackBasedSubscriber.onFailure(Subscribers.java:93)
                ... 28 more
Caused by: dev.ai4j.openai4j.OpenAiHttpException: {"object":"error","message":"[{'type': 'extra_forbidden', 'loc': ('body', 'tools'), 'msg': 'Extra inputs are not permitted', 'input': [{'type': 'function', 'function': {'name': 'updateClaimStatus', 'description': 'update claim status', 'parameters': {'type': 'object', 'properties': {'claimId': {'type': 'integer'}, 'status': {'type': 'string'}}, 'required': ['claimId', 'status']}}}]}]","type":"BadRequestError","param":null,"code":400}
        at io.quarkiverse.langchain4j.openai.OpenAiRestApi.toException(OpenAiRestApi.java:170)
        at io.quarkiverse.langchain4j.openai.OpenAiRestApi_toException_ResponseExceptionMapper_f35c1c86580504f69920f9de921a22bd696c020f.toThrowable(Unknown Source)
        at io.quarkus.rest.client.reactive.runtime.MicroProfileRestClientResponseFilter.filter(MicroProfileRestClientResponseFilter.java:49)
        at org.jboss.resteasy.reactive.client.handlers.ClientResponseFilterRestHandler.handle(ClientResponseFilterRestHandler.java:21)
        at org.jboss.resteasy.reactive.client.handlers.ClientResponseFilterRestHandler.handle(ClientResponseFilterRestHandler.java:10)
        at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.invokeHandler(AbstractResteasyReactiveContext.java:231)
        at org.jboss.resteasy.reactive.common.core.AbstractResteasyReactiveContext.run(AbstractResteasyReactiveContext.java:147)
        ... 12 more

[edeandrea]

I need to read/play with it a bit more. I've done some searching and it should be possible. At least thats what I think.

But I've been known to be wrong on occasion.... :D

[geoand]

If you can make it work, that would be nice!

I've never tried to make it work myself, but just the thought of it makes me not even want to try 😜

[edeandrea]

I've never tried to make it work myself, but just the thought of it makes me not even want to try

I'm definitely there with you as I'm digging through it, just trying to watch things fly by...