search

quarkiverse / quarkus-langchain4j

Quarkus Langchain4j extension
https://docs.quarkiverse.io/quarkus-langchain4j/dev/index.html
Apache License 2.0
149 stars 89 forks source link

Issue with using @SystemMessage(fromResource=...) in dev mode. #618

Open fmatar opened 6 months ago

fmatar commented 6 months ago

I have moved my prompt content to a file named my_prompt.txt to resources/dev/langchain4j/services

My service looks as follows:

interface MyService {
  @SystemMessage(fromResource = "my_prompt.txt")
  String chat();
}

The following error happens in dev mode only, I can get around it by building and running it in prod, not fun!

2024-05-23 09:36:27,410 ERROR [io.qua.dep.dev.IsolatedDevModeMain] (Aesh InputStream Reader) Failed to start quarkus: java.lang.RuntimeException: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
        [error]: Build step io.quarkiverse.langchain4j.deployment.devui.LangChain4jDevUIProcessor#cardPage threw an exception: java.lang.NullPointerException: Cannot invoke "org.jboss.jandex.AnnotationValue.asStringArray()" because the return value of "org.jboss.jandex.AnnotationInstance.value()" is null
        at io.quarkiverse.langchain4j.deployment.devui.LangChain4jDevUIProcessor.lambda$addChatPage$4(LangChain4jDevUIProcessor.java:98)
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:212)
        at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:194)
        at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1709)
        at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:782)
        at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:291)
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:212)
        at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1709)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:556)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:546)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:622)
        at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:291)
        at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:631)
        at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:637)
        at java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:642)
        at io.quarkiverse.langchain4j.deployment.devui.LangChain4jDevUIProcessor.addChatPage(LangChain4jDevUIProcessor.java:99)
        at io.quarkiverse.langchain4j.deployment.devui.LangChain4jDevUIProcessor.cardPage(LangChain4jDevUIProcessor.java:51)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:849)
        at io.quarkus.builder.BuildContext.run(BuildContext.java:256)
        at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
        at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2516)
        at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2495)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1521)
        at java.base/java.lang.Thread.run(Thread.java:1570)
        at org.jboss.threads.JBossThread.run(JBossThread.java:483)

        at io.quarkus.runner.bootstrap.AugmentActionImpl.runAugment(AugmentActionImpl.java:337)
        at io.quarkus.runner.bootstrap.AugmentActionImpl.reloadExistingApplication(AugmentActionImpl.java:267)
        at io.quarkus.runner.bootstrap.AugmentActionImpl.reloadExistingApplication(AugmentActionImpl.java:60)
        at io.quarkus.deployment.dev.IsolatedDevModeMain.restartApp(IsolatedDevModeMain.java:190)
        at io.quarkus.deployment.dev.IsolatedDevModeMain.restartCallback(IsolatedDevModeMain.java:173)
        at io.quarkus.deployment.dev.RuntimeUpdatesProcessor.doScan(RuntimeUpdatesProcessor.java:542)
        at io.quarkus.deployment.console.ConsoleStateManager.forceRestart(ConsoleStateManager.java:175)
        at io.quarkus.deployment.console.ConsoleStateManager.lambda$installBuiltins$0(ConsoleStateManager.java:112)
        at io.quarkus.deployment.console.ConsoleStateManager$1.accept(ConsoleStateManager.java:77)
        at io.quarkus.deployment.console.ConsoleStateManager$1.accept(ConsoleStateManager.java:49)
        at io.quarkus.deployment.console.AeshConsole.lambda$setup$1(AeshConsole.java:278)
        at org.aesh.terminal.EventDecoder.accept(EventDecoder.java:118)
        at org.aesh.terminal.EventDecoder.accept(EventDecoder.java:31)
        at org.aesh.terminal.io.Decoder.write(Decoder.java:133)
        at org.aesh.readline.tty.terminal.TerminalConnection.openBlocking(TerminalConnection.java:216)
        at org.aesh.readline.tty.terminal.TerminalConnection.openBlocking(TerminalConnection.java:203)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1570)
Caused by: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
        [error]: Build step io.quarkiverse.langchain4j.deployment.devui.LangChain4jDevUIProcessor#cardPage threw an exception: java.lang.NullPointerException: Cannot invoke "org.jboss.jandex.AnnotationValue.asStringArray()" because the return value of "org.jboss.jandex.AnnotationInstance.value()" is null
        at io.quarkiverse.langchain4j.deployment.devui.LangChain4jDevUIProcessor.lambda$addChatPage$4(LangChain4jDevUIProcessor.java:98)
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:212)
        at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:194)
        at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1709)
        at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:782)
        at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:291)
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:212)
        at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1709)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:556)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:546)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:622)
        at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:291)
        at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:631)
        at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:637)
        at java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:642)
        at io.quarkiverse.langchain4j.deployment.devui.LangChain4jDevUIProcessor.addChatPage(LangChain4jDevUIProcessor.java:99)
        at io.quarkiverse.langchain4j.deployment.devui.LangChain4jDevUIProcessor.cardPage(LangChain4jDevUIProcessor.java:51)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:849)
        at io.quarkus.builder.BuildContext.run(BuildContext.java:256)
        at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
        at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2516)
        at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2495)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1521)
        at java.base/java.lang.Thread.run(Thread.java:1570)
        at org.jboss.threads.JBossThread.run(JBossThread.java:483)

        at io.quarkus.builder.Execution.run(Execution.java:123)
        at io.quarkus.builder.BuildExecutionBuilder.execute(BuildExecutionBuilder.java:79)
        at io.quarkus.deployment.QuarkusAugmentor.run(QuarkusAugmentor.java:160)
        at io.quarkus.runner.bootstrap.AugmentActionImpl.runAugment(AugmentActionImpl.java:333)
        ... 18 more
Caused by: java.lang.NullPointerException: Cannot invoke "org.jboss.jandex.AnnotationValue.asStringArray()" because the return value of "org.jboss.jandex.AnnotationInstance.value()" is null
        at io.quarkiverse.langchain4j.deployment.devui.LangChain4jDevUIProcessor.lambda$addChatPage$4(LangChain4jDevUIProcessor.java:98)
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipelin212)va:
        at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:194)
        at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1709)
        at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:782)
        at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:291)
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:212)
        at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1709)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:556)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:546)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:622)
        at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:291)
        at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:631)
        at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:637)
        at java.base/java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:642)
        at io.quarkiverse.langchain4j.deployment.devui.LangChain4jDevUIProcessor.addChatPage(LangChain4jDevUIProcessor.java:99)
        at io.quarkiverse.langchain4j.deployment.devui.LangChain4jDevUIProcessor.cardPage(LangChain4jDevUIProcessor.java:51)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:849)
        at io.quarkus.builder.BuildContext.run(BuildContext.java:256)
        at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
        at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2516)
        at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2495)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1521)
        at java.base/java.lang.Thread.run(Thread.java:1570)
        at org.jboss.threads.JBossThread.run(JBossThread.java:483)
fmatar commented 6 months ago

Looking at the issue I think it's quite an easy fix:

https://github.com/quarkiverse/quarkus-langchain4j/blob/2ac7810478c27424a7ae0e47ae5d4ae888a6ade7/core/deployment/src/main/java/io/quarkiverse/langchain4j/deployment/devui/LangChain4jDevUIProcessor.java#L97

The filtering could include the non null values. Possibly update 

    private void addChatPage(CardPageBuildItem card, List<DeclarativeAiServiceBuildItem> aiServices) {
        List<String> systemMessages = aiServices.stream()
                .map(s -> s.getServiceClassInfo())
                .flatMap(c -> c.annotations().stream()) //This includes method annotations
                .filter(a -> a.name().equals(LangChain4jDotNames.SYSTEM_MESSAGE))
                .map(a -> String.join("", a.value().asStringArray()))
                .toList();
        ...
    }

to:

    private void addChatPage(CardPageBuildItem card, List<DeclarativeAiServiceBuildItem> aiServices) {
        List<String> systemMessages = aiServices.stream()
                .map(s -> s.getServiceClassInfo())
                .flatMap(c -> c.annotations().stream()) //This includes method annotations
                .filter(a -> a.name().equals(LangChain4jDotNames.SYSTEM_MESSAGE) && a.value()!=null)
                .map(a -> String.join("", a.value().asStringArray()))
                .toList();
        ...
    }

Thoughts?

gsmet commented 6 months ago

I think you would have to resolve the system message instead and reading it from the file. Not sure where it's done for runtime but you need something similar (or a way to propagate the value) in Dev UI.

fmatar commented 6 months ago

I'm looking a bit more into it, thank you for the suggestion

geoand commented 6 months ago

cc @jmartisk

fmatar commented 6 months ago

found a super hacky way to make it work. 

@SystemMessage(value="", fromResource="my_prompt.txt")

fromResource still overrides the value.

fmatar commented 6 months ago

That also uncovered another issue. In dev mode a message changed from an external file won't be changed unless I rebuild the app

jmartisk commented 6 months ago

That also uncovered another issue. In dev mode a message changed from an external file won't be changed unless I rebuild the app

Yeah the extension will have to declare that file to be watched for changes, this should be doable. Sorry for the delay in answering, I'm traveling all week and on wifi that mostly doesn't work, but I'll have a look into this as soon as I can.

geoand commented 6 months ago

this should be doable.

Definitely doable and easy 😜

geoand commented 6 months ago

639 takes care of the watches resource files issue