search

quarkiverse / quarkus-langchain4j

Quarkus Langchain4j extension
https://docs.quarkiverse.io/quarkus-langchain4j/dev/index.html
Apache License 2.0
133 stars 79 forks source link

SystemMessage too large. #635

Open fmatar opened 4 months ago

fmatar commented 4 months ago 
  @SystemMessage(value = "", fromResource = "large_prompt.txt")
  String chat(@MemoryId String cid, @UserMessage String userMessage);

If the file is large (mine is around 87kb) I get the following exception.

io.vertx.core.impl.NoStackTraceException Caused by: java.lang.RuntimeException: io.quarkus.builder.BuildException: Build failure: Build failed due to errors [error]: Build step io.quarkus.deployment.steps.MainClassBuildStep#build threw an exception: java.lang.RuntimeException: Failed to record call to method public void io.quarkiverse.langchain4j.runtime.AiServicesRecorder.setMetadata(java.util.Map) at io.quarkus.deployment.recording.BytecodeRecorderImpl.writeBytecode(BytecodeRecorderImpl.java:480) at io.quarkus.deployment.steps.MainClassBuildStep.writeRecordedBytecode(MainClassBuildStep.java:501) at io.quarkus.deployment.steps.MainClassBuildStep.build(MainClassBuildStep.java:201) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) at java.base/java.lang.reflect.Method.invoke(Method.java:580) at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:849) at io.quarkus.builder.BuildContext.run(BuildContext.java:256) at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18) at org.jboss.threads.EnhancedQueueExecutor$Task.doRunWith(EnhancedQueueExecutor.java:2516) at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2495) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1521) at java.base/java.lang.Thread.run(Thread.java:1570) at org.jboss.threads.JBossThread.run(JBossThread.java:483) Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.RuntimeException: String too large to record:

geoand commented 4 months ago

We should fix this, but I'll just note that using such a huge prompt as a system or user message is almost certainly not a good idea

jmartisk commented 4 months ago

Why is this being recorded at all, shouldn't the system message be overridable at runtime?

fmatar commented 4 months ago

I got around it by setting the SystemPrompt to a variable.

@SystemMessage="{system_prompt}"

And then in my @PostConstruct I'm setting the prompt as a global variable.

Not the usual approach for doing thing however this specific prompt is dictated by a massive schema definition.

geoand commented 3 months ago

Why is this being recorded at all, shouldn't the system message be overridable at runtime?

No, it's not