bug: Mistral streaming mode raise a NPE if content JSON field is missing

[philippart-s]

Hi, I use Mistral 7b model self deployed and I have the following error in streaming mode:

java.lang.NullPointerException: `emit` called with `null`.
        at io.smallrye.mutiny.operators.multi.builders.SerializedMultiEmitter.emit(SerializedMultiEmitter.java:138)
        at dev.langchain4j.service.AiServiceStreamingResponseHandler.onNext(AiServiceStreamingResponseHandler.java:53)
        at io.quarkiverse.langchain4j.mistralai.QuarkusMistralAiClient$2.accept(QuarkusMistralAiClient.java:82)
        at io.quarkiverse.langchain4j.mistralai.QuarkusMistralAiClient$2.accept(QuarkusMistralAiClient.java:76)

this seems to be due to the fact that the first JSON response line has not a content field:

- body: data: {"id":"cmpl-bb94685434ec46669cd161cc42ee5ed5","model":"Mixtral-8x7B-Instruct-v0.1","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]}

data: {"id":"cmpl-bb94685434ec46669cd161cc42ee5ed5","object":"chat.completion.chunk","created":1713899355,"model":"Mixtral-8x7B-Instruct-v0.1","choices":[{"index":0,"delta":{"role":null,"content":"Certain"},"finish_reason":null}]}

...

Perhaps it should be fine to test the presence of the content field?

[geoand]

@langchain4j is this a known issue on the library side?

[langchain4j]

@geoand I can't reproduce the missing content scenario, but it should work fine on our side:

Handling of missing content: https://github.com/langchain4j/langchain4j/blob/d34e1df9caa7747083eb927cb92283d2cc37c9e0/langchain4j-mistral-ai/src/main/java/dev/langchain4j/model/mistralai/DefaultMistralAiClient.java#L139

Test: https://github.com/langchain4j/langchain4j/blob/d34e1df9caa7747083eb927cb92283d2cc37c9e0/langchain4j/src/test/java/dev/langchain4j/service/StreamingAiServicesIT.java#L48

[philippart-s]

if I can add my opinion, for me the issue is here: https://github.com/quarkiverse/quarkus-langchain4j/blob/9e6f7596e23888a6e5d909ff9d5f6d086be65e5a/mistral/runtime/src/main/java/io/quarkiverse/langchain4j/mistralai/QuarkusMistralAiClient.java#L82

If the chunk is null it is sent to this: https://github.com/langchain4j/langchain4j/blob/d34e1df9caa7747083eb927cb92283d2cc37c9e0/langchain4j/src/main/java/dev/langchain4j/service/AiServiceStreamingResponseHandler.java#L53

And then a NPE is thrown.

[geoand]

@philippart-s would you like to provide a PR with the proposed fix?

[philippart-s]

@philippart-s would you like to provide a PR with the proposed fix?

If you agree with my proposal, I'd love to 😉

[geoand]

Go right ahead :)

[nhh1603]

I'm not sure if it's the same problem, but I also use Mistral 7b locally, and the problem is each time it finishes token streaming, it returns this error to me. When i inspect with Postman it seems to me that the last token has no content in it (to indicate that it's done) but somehow the code continues to pass in the onNext block which causes this IllegalArgumentException instead of entering the onComplete block. For the streaming code, I literally copied paste the one from the example.

[geoand]

Have you tried with 0.12.1?

[nhh1603]

That is the version of which one ?

[geoand]

Of quarkus-langchain4j-mistral-ai

[nhh1603]

Ok thanks I'll try it out and I'll let you know if the problem persists

[geoand]

🙏🏼