search

quarkiverse / quarkus-langchain4j

Quarkus Langchain4j extension
https://docs.quarkiverse.io/quarkus-langchain4j/dev/index.html
Apache License 2.0
138 stars 81 forks source link

Use llama3.1 with ollama? #787

Closed edeandrea closed 2 months ago

edeandrea commented 2 months ago

I can't seem to be able to use llama3.1 with the ollama extension. If I set quarkus.langchain4j.ollama.parasol-chat.model-id=llama3.1, when my app starts up I see 

Ollama model pull: 2024-07-26 09:29:22,744 INFO  [io.qua.lan.dep.dev.DevServicesOllamaProcessor] (build-26) Preloading model llama3

and all the requests target llama3, even though llama3.1 has been pulled

╰─ ollama ls
NAME                            ID              SIZE    MODIFIED       
llama3:latest                   365c0bd3c000    4.7 GB  20 minutes ago  
llama3.1:latest                 62757c860e01    4.7 GB  22 minutes ago  
mxbai-embed-large:latest        468836162de7    669 MB  24 hours ago    
nomic-embed-text:latest         0a109f422b47    274 MB  7 weeks ago     
mistral:latest                  2ae6f6dd7a3d    4.1 GB  7 weeks ago

See the request in the log:

2024-07-26 09:31:20,207 INFO  [io.qua.lan.oll.OllamaRestApi$OllamaLogger] (vert.x-eventloop-thread-0) Request:,0.6377601623535156,1.3655022382736206,0.37206006050109863,-0.5449820756912231,-0.3099823594093323,-0.03494488075375557,-0.76306790113
- method: POST
- url: http://localhost:11434/api/chat
- headers: [Accept: text/event-stream], [Content-Type: application/json], [User-Agent: Quarkus REST Client], [content-length: 4481]
- body: {
  "model" : "llama3",
  "messages" : [ {
    "role" : "system",
    "content" : "You are a helpful, respectful and honest assistant named \"Parasol Assistant\".\nYou will be given a claim summary, references to provide you with information, and a question. You must answer the question based as much as possible on this claim with the help of the references.\nAlways answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.\n\nIf a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.\n"
  }, {
    "role" : "user",
    "content" : "    Claim Summary:\n    \n    On January 2nd, 1955, at around 3:30 PM, a car accident occurred at the intersection of Colima Road and Azusa Avenue in Hill Vallet. The involved parties were Marty McFly, driving a silver Delorean DMC-12 (OUTA-TIME), and Biff Tanner in a blue Type 2 Volkswagen Bus (BIF-RULZ).\n\n    Marty was heading south on Colima Road when Biff failed to stop at the red traffic signal on Asuza Avenue, causing a collision with Marty's vehicle. Both drivers exchanged information and took photos of the accident scene, which included damages to the front driver and passenger side of Marty's Delorean DMC-12 and the front driver's side of Biff's Volkswagen Bus. No injuries were reported.\n\n    Marty has attached necessary documents, such as photos, a police report, and an estimate for repair costs, to his email. He requests prompt attention to the claim and is available at (916) 555-4385 or marty.mcfly@email.com for any additional information or documentation needed.\n    \n\n    Question: Should this claim be covered?\n\n\nAnswer using the following information:\n5.\n\nPremiums and Deductibles\n5.1 Premiums are calculated based on factors including driver history, vehicle type, and\n\ncoverage selected.\n5.2 Deductibles apply to collision and comprehensive coverage as specified on the\n\ndeclarations page.\n\n6. Cancellation\n6.1 The policyholder may cancel at any time with pro-rata refund of premiums.\n6.2 The insurer reserves the right to cancel with 30 days' notice, or immediately in cases of\n\nnon-payment or fraud.\n\n7. Definitions\n7.1 \"Insured vehicle\" refers to the automobile listed on the declarations page.\n7.2 \"Accident\" means a sudden, unexpected event resulting in damage or injury.\n\n8. Additional Provisions\n8.1 Coverage extends to other drivers listed on the policy.\n8.2 Rental car coverage may be included if specified in the policy.\n8.3 Roadside assistance may be available if selected as an add-on.\n\n9. Contact Information\nFor claims or inquiries:\nPhone: 1-800-CAR-SAFE\nEmail: claims@parasol.com\n\nPolicy Number: #AUTO-67890-2024\nPolicyholder: [Name]\nEffective Date: [Date]\n\nParasol Insurance Co.\nPolicy Document\n\n1. Coverage\n1.1 This policy provides coverage for the insured vehicle as specified in the declarations page.\n1.2 Coverage types may include:\n\na) Liability (Bodily Injury and Property Damage)\nb) Collision\nc) Comprehensive\nd) Personal Injury Protection (PIP)\ne) Uninsured/Underinsured Motorist\n\n1.3 Specific coverage limits and deductibles are listed on the declarations page.\n\n2. Policy Term\n2.1 The policy term is six months from the date of inception, unless otherwise specified.\n2.2 Renewal is not automatic and must be initiated by the policyholder.\n\n3. Exclusions\n3.1 Damage caused by intentional acts or gross negligence is not covered.\n3.2 Normal wear and tear is excluded from coverage.\n3.3 Damage resulting from racing or using the vehicle for commercial purposes without proper\n\nendorsement is excluded.\n3.4 Driving under the influence of alcohol or drugs voids coverage for that incident.\n\n4. Claim Process\n4.1 Claims should be filed as soon as possible after an incident.\n4.2 Policyholders must provide all relevant information, including police reports when\n\napplicable.\n4.3 An adjuster may be assigned to inspect the vehicle and assess damages.\n\n5. Premiums and Deductibles\n5.1 Premiums are calculated based on factors including driver history, vehicle type, and"
  } ],
  "options" : {
    "temperature" : 0.3,
    "top_k" : 40,
    "top_p" : 0.9
  },
  "stream" : true
}
geoand commented 2 months ago

Can you please attach the sample you are using?

edeandrea commented 2 months ago

https://github.com/rh-rad-ai-roadshow/parasol-insurance/tree/private-docs-rag-solution/app

geoand commented 2 months ago

🙏🏼

geoand commented 2 months ago

We don't pull any models when the base-url has been set - it's same idea as with dev-services, meaning that if you configured something, we assume it's non something we should change

edeandrea commented 2 months ago

But even given that, I've already pulled it outside of quarkus. The requests to ollama are still specifying llama3 and not llama3.1

edeandrea commented 2 months ago

And you can see from the log it did pull llama3 (because it didn't already exist).

edeandrea commented 2 months ago

And in my config, I'm not setting a base url for ollama.

geoand commented 2 months ago

I'll have another look next week

edeandrea commented 2 months ago

Have a good weekend!

geoand commented 2 months ago

You too!

On Fri, Jul 26, 2024, 18:57 Eric Deandrea @.***> wrote:

Have a good weekend!

— Reply to this email directly, view it on GitHub https://github.com/quarkiverse/quarkus-langchain4j/issues/787#issuecomment-2253050868, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABBMDP2ZSBTA65XC73P6J3DZOJWWHAVCNFSM6AAAAABLQRAC7GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJTGA2TAOBWHA . You are receiving this because you commented.Message ID: @.***>

edeandrea commented 2 months ago

What's interesting is that I tried to upgrade to 0.17.0.CR1 and now it does pre-load and use the llama3.1 model....

╰─ ollama ps           
NAME                    ID              SIZE    PROCESSOR       UNTIL              
llama3.1:latest         62757c860e01    6.7 GB  100% GPU        4 minutes from now      
nomic-embed-text:latest 0a109f422b47    1.1 GB  100% GPU        2 minutes from now

BUT....

Now my embedding seems to be broken when I ask the chat bot a question (This works fine in 0.16.4)...

2024-07-26 12:44:00,002 ERROR [io.qua.web.nex.run.Endpoints] (vert.x-eventloop-thread-1) Unable to close connection [607941c7-297e-4fad-8456-6d2dd5e29f11] due to unhandled failure [java.net.SocketException: Connection reset]: io.netty.channel.StacklessClosedChannelException
2024-07-26 12:44:00,003 INFO  [org.par.res.ClaimWebsocketChatBot] (vert.x-worker-thread-2) Websocket connection 607941c7-297e-4fad-8456-6d2dd5e29f11 closed
2024-07-26 12:44:09,019 ERROR [io.qua.lan.run.ais.AiServiceMethodImplementationSupport] (vert.x-eventloop-thread-0) Execution of org.parasol.ai.ClaimService#chat failed: org.jboss.resteasy.reactive.common.core.BlockingNotAllowedException
        at org.jboss.resteasy.reactive.client.impl.InvocationBuilderImpl.unwrap(InvocationBuilderImpl.java:199)
        at org.jboss.resteasy.reactive.client.impl.InvocationBuilderImpl.method(InvocationBuilderImpl.java:328)
        at io.quarkiverse.langchain4j.ollama.OllamaRestApi$$QuarkusRestClientInterface.embeddings(Unknown Source)
        at io.quarkiverse.langchain4j.ollama.OllamaClient.embedding(OllamaClient.java:44)
        at io.quarkiverse.langchain4j.ollama.OllamaEmbeddingModel.lambda$embedAll$0(OllamaEmbeddingModel.java:36)
        at java.base/java.util.Collections$SingletonList.forEach(Collections.java:5186)
        at io.quarkiverse.langchain4j.ollama.OllamaEmbeddingModel.embedAll(OllamaEmbeddingModel.java:30)
        at dev.langchain4j.model.embedding.EmbeddingModel.embed(EmbeddingModel.java:34)
        at dev.langchain4j.model.embedding.EmbeddingModel.embed(EmbeddingModel.java:24)
        at dev.langchain4j.model.embedding.EmbeddingModel_OfFfdXYI2wr9Ymzsx9IXE32kX_M_Synthetic_ClientProxy.embed(Unknown Source)
        at dev.langchain4j.rag.content.retriever.EmbeddingStoreContentRetriever.retrieve(EmbeddingStoreContentRetriever.java:179)
        at dev.langchain4j.rag.DefaultRetrievalAugmentor.process(DefaultRetrievalAugmentor.java:174)
        at dev.langchain4j.rag.DefaultRetrievalAugmentor.augment(DefaultRetrievalAugmentor.java:154)
        at dev.langchain4j.rag.DefaultRetrievalAugmentor.augment(DefaultRetrievalAugmentor.java:140)
        at io.quarkiverse.langchain4j.easyrag.runtime.EasyRetrievalAugmentor.augment(EasyRetrievalAugmentor.java:31)
        at dev.langchain4j.rag.RetrievalAugmentor.augment(RetrievalAugmentor.java:39)
        at io.quarkiverse.langchain4j.easyrag.runtime.EasyRetrievalAugmentor_VxHnKxDROyS2BgWURtZmLL_kruM_Synthetic_ClientProxy.augment(Unknown Source)
        at io.quarkiverse.langchain4j.runtime.aiservice.AiServiceMethodImplementationSupport.doImplement(AiServiceMethodImplementationSupport.java:136)
        at io.quarkiverse.langchain4j.runtime.aiservice.AiServiceMethodImplementationSupport.implement(AiServiceMethodImplementationSupport.java:102)
        at org.parasol.ai.ClaimService$$QuarkusImpl.chat(Unknown Source)
        at org.parasol.ai.ClaimService$$QuarkusImpl_ClientProxy.chat(Unknown Source)
        at org.parasol.resources.ClaimWebsocketChatBot.onMessage(ClaimWebsocketChatBot.java:36)
        at org.parasol.resources.ClaimWebsocketChatBot_WebSocketServerEndpoint.doOnTextMessage(Unknown Source)
        at io.quarkus.websockets.next.runtime.WebSocketEndpointBase.doExecute(WebSocketEndpointBase.java:175)
        at io.quarkus.websockets.next.runtime.WebSocketEndpointBase$1.run(WebSocketEndpointBase.java:112)
        at io.quarkus.websockets.next.runtime.ConcurrencyLimiter.run(ConcurrencyLimiter.java:53)
        at io.quarkus.websockets.next.runtime.WebSocketEndpointBase.execute(WebSocketEndpointBase.java:109)
        at io.quarkus.websockets.next.runtime.WebSocketEndpointBase.onTextMessage(WebSocketEndpointBase.java:77)
        at io.quarkus.websockets.next.runtime.Endpoints.lambda$initialize$1(Endpoints.java:120)
        at io.quarkus.websockets.next.runtime.Endpoints$5$1.handle(Endpoints.java:319)
        at io.quarkus.websockets.next.runtime.Endpoints$5$1.handle(Endpoints.java:316)
        at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:279)
        at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:261)
        at io.vertx.core.impl.ContextInternal.lambda$runOnContext$0(ContextInternal.java:59)
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:1583)

2024-07-26 12:44:09,073 INFO  [org.par.res.ClaimWebsocketChatBot] (vert.x-worker-thread-2) Websocket connection 18f43077-9427-4e16-8563-5d130db1630e closed
2024-07-26 12:44:09,089 INFO  [io.qua.lan.oll.OllamaRestApi$OllamaLogger] (vert.x-eventloop-thread-0) Request:
- method: POST
- url: http://localhost:11434/api/embeddings
- headers: [Accept: application/json], [Content-Type: application/json], [User-Agent: Quarkus REST Client], [content-length: 1164]
- body: {
  "model" : "nomic-embed-text",
  "prompt" : "    Claim Summary:\n    \n    On January 2nd, 1955, at around 3:30 PM, a car accident occurred at the intersection of Colima Road and Azusa Avenue in Hill Vallet. The involved parties were Marty McFly, driving a silver Delorean DMC-12 (OUTA-TIME), and Biff Tanner in a blue Type 2 Volkswagen Bus (BIF-RULZ).\n\n    Marty was heading south on Colima Road when Biff failed to stop at the red traffic signal on Asuza Avenue, causing a collision with Marty's vehicle. Both drivers exchanged information and took photos of the accident scene, which included damages to the front driver and passenger side of Marty's Delorean DMC-12 and the front driver's side of Biff's Volkswagen Bus. No injuries were reported.\n\n    Marty has attached necessary documents, such as photos, a police report, and an estimate for repair costs, to his email. He requests prompt attention to the claim and is available at (916) 555-4385 or marty.mcfly@email.com for any additional information or documentation needed.\n    \n\n    Question: Does the policy cover this claim?\n\nYou must answer strictly in the following JSON format: {\n}"
}

2024-07-26 12:44:09,192 INFO  [io.qua.lan.oll.OllamaRestApi$OllamaLogger] (vert.x-eventloop-thread-0) Response:
- status code: 200
- headers: [Content-Type: application/json; charset=utf-8], [Date: Fri, 26 Jul 2024 16:44:09 GMT], [Transfer-Encoding: chunked]
- body: {"embedding":[-0.5406284332275391,0.03225094825029373,-3.108903646469116,-0.40123674273490906,1.4819371700286865,-0.2436724156141281,0.3510435223579407,0.9201043844223022,0.573516845703125,-1.086242914199829,-0.7788794040679932,-0.30110228061676025,0.8159918189048767,0.3948259949684143,0.37102729082107544,-1.300229787826538,0.17488649487495422,-0.875462532043457,-0.8818553686141968,0.3128630220890045,-0.09109801054000854,0.16480238735675812,-2.0849685668945312,0.9108506441116333,2.50067138671875,-0.9221913814544678,0.008729502558708191,0.6555370688438416,-0.9782894253730774,-0.17205344140529633,1.1016409397125244,-0.3670438528060913,-0.7018741369247437,-1.5449466705322266,-1.8335007429122925,-0.7774045467376709,0.313204824924469,0.28014227747917175,-0.38681378960609436,0.38690608739852905,2.061419725418091,-0.3383425176143646,0.17885948717594147,-0.4176533818244934,0.30500733852386475,0.16479095816612244,1.1857030391693115,-0.37430816888809204,0.509952962398529,0.7151135206222534,0.08446626365184784,0.16568228602409363,0.11806389689445496,-0.5772643089294434,0.118152916431427,-0.6420964002609253,0.05821719020605087,1.2402126789093018,-0.3862432837486267,0.4839318096637726,1.0643184185028076,1.1555390357971191,0.16133412718772888,0.5959079265594482,0.05433379486203194,-0.7411357164382935,-0.1119714230298996,1.4443869590759277,0.5686404705047607,-1.3989787101745605,0.23834240436553955,0.6049525737762451,-0.5713115930557251,0.22065092623233795,-0.7871074676513672,1.5348550081253052,-1.1107478141784668,-0.24212297797203064,0.338199257850647,0.9919568300247192,0.3354780972003937,0.5592159628868103,0.6361432075500488,-0.0071087125688791275,0.5169007778167725,0.14606249332427979,0.49928703904151917,0.0025122053921222687,-0.7022178173065186,1.0684711933135986,0.9792764186859131,0.2741371989250183,0.8238548040390015,-0.22025492787361145,-0.1881219893693924,-0.11624547839164734,-1.0358014106750488,-0.9305236339569092,-0.6624308824539185,0.21400053799152374,-0.6026293039321899,-1.183587908744812,0.08793789893388748,0.46455997228622437,0.24649879336357117,1.1797958612442017,-0.48481014370918274,0.42967528104782104,0.2591245770454407,-1.4050968885421753,-0.73763507604599,-0.13631322979927063,-0.03960280120372772,-0.8577829599380493,0.14537197351455688,0.5341999530792236,0.5816956758499146,0.014539778232574463,-0.1776331067085266,0.7342529892921448,-1.2309072017669678,0.32736003398895264,-0.8597052097320557,1.441633701324463,0.024519555270671844,-0.1322270929813385,-1.3408488035202026,0.5936336517333984,-0.196164071559906,0.1660243272781372,-1.424464225769043,-0.35283005237579346,0.704225480556488,0.8172281980514526,0.055702660232782364,0.45885276794433594,-0.03786194697022438,0.2514815926551819,-0.5405761003494263,0.8787182569503784,1.4036948680877686,0.5533261299133301,-0.8253779411315918,-0.5314430594444275,0.493571013212204,-0.5593346357345581,-0.12559902667999268,-0.10460729151964188,-0.11920762062072754,-0.6352040767669678,-0.9258774518966675,-0.667914628982544,0.11140275001525879,0.20206724107265472,0.19841036200523376,-0.5101191401481628,-0.29715150594711304,-0.5608383417129517,-0.43179720640182495,1.1293325424194336,0.43546104431152344,-0.2644696533679962,-0.7582123279571533,1.4527161121368408,-0.3135107457637787,-0.2139986753463745,0.07231775671243668,0.28037694096565247,0.3123375177383423,0.6233230829238892,0.2579854130744934,-1.248634934425354,0.0053656091913580894,-0.2196296602487564,0.4757385849952698,-1.1974915266036987,-1.2515809535980225,-1.2500255107879639,0.7331627607345581,-0.48071447014808655,1.4909167289733887,-1.0724149942398071,1.4523898363113403,0.7697033286094666,-0.25841736793518066,-1.3701252937316895,0.2127397209405899,-0.4789942800998688,-0.048509299755096436,-0.697618842124939,-0.40985941886901855,0.514511227607727,-1.3190677165985107,-1.4282267093658447,-0.7307229042053223,-0.8669058680534363,1.035813570022583,0.007136759348213673,-0.11334153264760971,-0.9523705840110779,-0.8853991031646729,0.007148735225200653,-0.2662345767021179,0.1598621904850006,-0.6238612532615662,0.12375783920288086,0.24386586248874664,0.7481091022491455,-0.2157876342535019,-0.12583467364311218,0.8453352451324463,-0.2848062515258789,-0.5622520446777344,0.36867380142211914,-0.8409654498100281,0.07409090548753738,-0.12286286056041718,0.07232052087783813,0.8587687015533447,0.0066047124564647675,0.03841511532664299,0.02892322838306427,0.3374844789505005,-0.3789498507976532,0.23620590567588806,0.1330079287290573,-0.43664225935935974,-1.1975982189178467,-1.4009233713150024,-0.3923547863960266,0.306652307510376,-1.3850419521331787,0.2652170956134796,0.26762449741363525,0.23179465532302856,0.597240686416626,0.2444700449705124,0.7458035349845886,1.1444497108459473,-0.4687323570251465,-0.15894776582717896,0.5703766345977783,-0.1354835331439972,0.40439239144325256,-0.4261319637298584,0.30025970935821533,-0.2639717757701874,-0.8504904508590698,1.182941198348999,2.137838840484619,0.07503052055835724,-0.038211844861507416,0.07119196653366089,-0.3154640197753906,0.23444922268390656,-1.2944037914276123,0.10821978747844696,0.19305990636348724,-0.7097557783126831,0.9506368637084961,0.44535911083221436,-1.1116715669631958,0.47841981053352356,-1.2472808361053467,-1.3908398151397705,-0.6933908462524414,-0.42651844024658203,-0.510668158531189,0.398775577545166,-0.8511786460876465,-0.16618986427783966,-0.4321768283843994,-0.6784273386001587,1.335806965827942,0.21489626169204712,0.05615288019180298,-0.8945020437240601,-1.0065293312072754,-0.8343916535377502,0.0790410190820694,0.0452498123049736,-0.6564627885818481,0.2811098098754883,-0.08534544706344604,0.715383768081665,0.45484060049057007,-0.20722508430480957,0.8563885688781738,0.3323425054550171,0.24946695566177368,0.27802062034606934,0.5180683135986328,-0.07674939930438995,1.8163046836853027,-0.5023161172866821,1.0039331912994385,1.0360716581344604,-0.4014527499675751,-0.32550814747810364,-1.3769769668579102,0.30069345235824585,0.6508768796920776,0.25602030754089355,0.6654205918312073,-0.12281568348407745,-0.17455816268920898,-0.16936630010604858,0.21891680359840393,0.5890121459960938,0.3800962567329407,-0.28346899151802063,-0.2620090842247009,-0.6074776649475098,-0.34614160656929016,-0.19480279088020325,1.5129170417785645,0.6719975471496582,0.5335121154785156,0.8400816321372986,1.4137357473373413,-0.3337923586368561,-0.2729839086532593,-1.0550967454910278,-0.36003610491752625,-0.050228752195835114,0.8181229829788208,-0.9615192413330078,0.16698595881462097,1.1253842115402222,-0.028270993381738663,0.22639808058738708,0.3943272829055786,0.3987089991569519,-1.511918306350708,-0.021656887605786324,0.9452422857284546,0.36947065591812134,0.28581351041793823,-0.15817119181156158,0.7729723453521729,0.9760687351226807,-0.47957342863082886,-0.4790347218513489,-1.6021397113800049,-0.4437658190727234,-0.7004001140594482,-0.8602754473686218,0.38566428422927856,0.8090289831161499,0.8850829601287842,-0.08682981133460999,-0.5335105061531067,-0.5989158153533936,0.009654348716139793,0.4509119689464569,-0.11203902959823608,-0.10740678012371063,0.058900561183691025,0.008183639496564865,1.2700567245483398,1.2380552291870117,0.07756214588880539,0.31046247482299805,-0.015200162306427956,0.8758162260055542,0.6297993659973145,0.6314309239387512,-0.527402937412262,0.631972074508667,-0.09172471612691879,-0.7031792402267456,-0.43795478343963623,-0.27869856357574463,0.387445867061615,0.7519376873970032,-0.2009812891483307,-0.2522347569465637,-0.7222054600715637,-0.6992892026901245,1.378271460533142,0.6833740472793579,0.7456300854682922,0.18246722221374512,0.4742451310157776,0.871292233467102,-0.4478362202644348,0.14459636807441711,-0.3737369477748871,0.24886831641197205,0.8849540948867798,-0.708189070224762,-0.31580087542533875,-1.1074965000152588,           -0.500076949596405,1.0113853216171265,-0.7672843337059021,-0.22287800908088684,-0.023224540054798126,-0.5749038457870483,0.3108817934989929,-0.396679162979126,0.8175086379051208,-0.19182196259498596,-0.31357428431510925,0.15165269374847412,-0.5865981578826904,-0.3335988223552704,-0.9116036891937256,0.6459240913391113,0.19953522086143494,-0.24403515458106995,0.9505124688148499,-0.1147007942199707,-1.2431282997131348,0.434121310710907,0.7995960712432861,0.7988415956497192,0.3798831105232239,-0.3036901354789734,-0.8225829005241394,0.5317501425743103,-0.04640473425388336,0.029483318328857422,-0.023109767585992813,0.4526793956756592,-0.4773022532463074,0.5939327478408813,0.39184585213661194,-0.3233022689819336,-0.4691739082336426,0.40173423290252686,0.347869336605072,-0.1854541301727295,0.6695389747619629,-0.6833854913711548,0.3482484221458435,0.5281304121017456,0.9120702743530273,1.299302577972412,0.021175269037485123,-0.5253118276596069,-1.2182769775390625,0.5539865493774414,-0.5469126105308533,1.4702109098434448,1.7237567901611328,1.0943784713745117,-1.222655177116394,0.17754369974136353,1.2904596328735352,0.10116907954216003,0.07210449874401093,0.17805638909339905,0.4234308898448944,0.7758051753044128,-1.2437148094177246,0.22852841019630432,-0.12232738733291626,0.3260379433631897,0.8702121376991272,0.14881347119808197,0.29461705684661865,0.6969854235649109,0.1270318180322647,-0.13629655539989471,-1.346144676208496,0.2511422038078308,0.04799581319093704,-0.3597889244556427,1.202315092086792,-1.3712706565856934,-0.27652955055236816,0.016264688223600388,-0.6836543679237366,-0.3086485266685486,0.5902785062789917,0.1215028464794159,0.48854315280914307,0.8669346570968628,0.741576075553894,0.7411733865737915,-0.6952860355377197,-0.5904197692871094,-0.3497721552848816,0.47406822443008423,0.39020881056785583,0.783639669418335,0.2715165615081787,0.5521088242530823,-0.5379135608673096,0.22608846426010132,-0.23505127429962158,-0.22202596068382263,-0.6249390840530396,0.09770818054676056,0.1710805594921112,-0.08109821379184723,1.0418460369110107,0.4851180911064148,0.9988039135932922,-0.002016749233007431,1.1235672235488892,0.44905102252960205,0.299447625875473,0.005324721336364746,-0.4601425528526306,0.052065491676330566,-0.0030734390020370483,-0.5197365283966064,0.031268395483493805,-0.4376315176486969,0.03715416043996811,-0.13516786694526672,0.2770996689796448,0.6821915507316589,-0.15443521738052368,-0.016023412346839905,0.22284401953220367,-0.10614029318094254,0.4907221496105194,0.19206847250461578,-1.007610559463501,0.2318052351474762,-0.06654760986566544,-1.2328321933746338,0.1360800415277481,0.45176634192466736,-1.3683537244796753,0.25269120931625366,0.20352979004383087,-0.2129654586315155,0.768917441368103,-0.20141008496284485,0.34165146946907043,-0.0964876189827919,0.37636202573776245,0.18806752562522888,0.0667550191283226,1.036351203918457,0.6167935729026794,-0.5544920563697815,-0.8086172342300415,0.039685554802417755,0.29771536588668823,0.20247882604599,-0.4420779347419739,0.9225658178329468,-0.5438093543052673,-0.0313861258327961,-0.7940955758094788,0.12071714550256729,-1.009594202041626,-0.5128965377807617,-0.7037822604179382,0.7494841814041138,-1.0132664442062378,-0.9635996222496033,-0.08457938581705093,0.26103639602661133,-0.6057567596435547,0.24373860657215118,-0.1288279891014099,0.636650562286377,0.9303805828094482,0.19601210951805115,0.3062134087085724,-0.21126452088356018,-0.725325345993042,-0.033144835382699966,-0.5216156244277954,0.787208616733551,-1.2115155458450317,0.43753015995025635,-0.3200514614582062,0.4192444086074829,0.27314493060112,0.30091917514801025,0.16744457185268402,-1.252871036529541,-1.4111385345458984,-0.6850239038467407,-0.7565140128135681,0.6329746246337891,0.7993878126144409,-0.11169350892305374,0.6660139560699463,-0.28909093141555786,-0.6081855297088623,0.011861708015203476,-0.1715480089187622,-0.07674795389175415,0.24158412218093872,1.2333987951278687,-0.29957762360572815,-0.8911685943603516,-0.2784840762615204,-0.29562699794769287,-0.22593334317207336,-0.6146622896194458,-0.5587435960769653,0.07949209213256836,0.0012989845126867294,0.5474397540092468,0.1716124415397644,0.19534465670585632,0.33517423272132874,-0.4397096037864685,-0.6594620943069458,0.2024085372686386,-1.2101495265960693,0.1329929232597351,-0.45121875405311584,-0.01445450633764267,0.10498916357755661,0.27358347177505493,0.024806756526231766,-0.18520291149616241,0.016385478898882866,-0.10566273331642151,0.685802698135376,-0.48539429903030396,-0.7843891382217407,0.80330491065979,-0.6302880048751831,-0.05700534209609032,-0.3836686611175537,-1.0686986446380615,-1.0919439792633057,-0.08289220929145813,1.0843456983566284,-0.3864976763725281,0.9377638101577759,-0.6459987759590149,-1.4594323635101318,-0.8989617824554443,1.066670298576355,0.46580150723457336,0.593161940574646,-0.18552173674106598,-0.49103373289108276,0.06111905351281166,0.16074815392494202,-0.8218235373497009,1.0157519578933716,0.38956552743911743,-0.6265138387680054,0.6368930339813232,0.8486602306365967,0.3988235890865326,-0.6887997388839722,2.0390846729278564,1.5370428562164307,0.18962933123111725,0.48630666732788086,0.6516808271408081,-1.1081595420837402,-0.2552342712879181,0.4580959379673004,-0.6935231685638428,-0.03194568678736687,0.20744165778160095,0.522781491279602,-0.6532588601112366,-1.0537351369857788,0.6816965341567993,0.2076716423034668,0.012908650562167168,0.5267552733421326,-1.4670064449310303,-0.1280699074268341,1.3615169525146484,0.4279959201812744,0.11186455190181732,-0.31951987743377686,0.07393461465835571,-0.31475400924682617,0.07599711418151855,1.3354997634887695,0.5213299989700317,-0.36407363414764404,0.6103230714797974,0.6004968285560608,0.15096169710159302,-0.44133710861206055,-0.2567834258079529,-1.862967848777771,0.04130125790834427,-0.670556366443634,0.2041337788105011,0.360950767993927,-0.200203076004982,-0.7106615900993347,0.26846128702163696,0.13009682297706604,0.518502950668335,-0.15332017838954926,-0.6973762512207031,0.4146677851676941,-0.5793535709381104,1.2202702760696411,0.07316712290048599,1.8536111116409302,0.21501773595809937,0.632166862487793,-0.26875925064086914,-0.6307892799377441,0.08492955565452576,-0.7666230201721191,-0.8291102051734924,-0.24306422472000122,1.2099344730377197,0.4498182237148285,-0.5311957001686096,0.5587023496627808,0.04017595201730728,-0.13592542707920074,0.1015012115240097,-0.9821344017982483,-0.18629735708236694,0.33141210675239563,1.3568601608276367,0.22958050668239594,-0.9262720346450806,-0.8183791041374207,-0.40092459321022034,-0.5938047170639038,0.2070881873369217,-0.6814700365066528,0.3456987738609314,-0.1213042140007019,-0.4144691824913025,0.20141339302062988,-0.9473404884338379,0.3234914541244507,-0.6702914237976074,-0.17534805834293365,-0.5182998776435852,-0.540135383605957,-0.24639390408992767,0.04448789358139038,-1.8538955450057983,-0.4179558753967285,0.32601532340049744,0.6352740526199341,-0.7314903736114502,-0.5117227435112,-0.642767608165741,0.5855831503868103,0.5890846252441406,-1.2665715217590332,1.1197426319122314,0.06929811090230942,-1.250552773475647,1.1436351537704468,0.47020286321640015,0.3431999087333679,0.6765674352645874,0.6587545871734619,1.6515181064605713,0.38674110174179077,-0.23116105794906616,-0.2837374806404114,-0.1260869950056076,-0.85990

It looks to me like for some reason easy rag is trying to call the blocking chat method on the OllamaRestApi, but the AIService in my case is streaming, so it should be using the streamingChat method.

geoand commented 2 months ago

What's interesting is that I tried to upgrade to 0.17.0.CR1 and now it does pre-load and use the llama3.1 model....

That's simply because we changed the default to llama3.1

geoand commented 2 months ago

@radcortez this seems to come down to an issue with ConfigMapping.

Essentially in the quarkus-langchain4jj-ollama extension I have:

@ConfigRoot(phase = BUILD_AND_RUN_TIME_FIXED)
@ConfigMapping(prefix = "quarkus.langchain4j.ollama")
public interface LangChain4jOllamaFixedRuntimeConfig {

    /**
     * Default model config.
     */
    @WithParentName
    OllamaConfig defaultConfig();

    /**
     * Named model config.
     */
    @ConfigDocSection
    @ConfigDocMapKey("model-name")
    @WithParentName
    @WithDefaults
    Map<String, OllamaConfig> namedConfig();

    interface OllamaConfig {

        /**
         * Chat model related settings
         */
        ChatModelFixedRuntimeConfig chatModel();
    }
}

@ConfigGroup
public interface ChatModelFixedRuntimeConfig {

    /**
     * Model to use. According to <a href="https://github.com/jmorganca/ollama/blob/main/docs/api.md#model-names">Ollama
     * docs</a>,
     * the default value is {@code llama3}
     */
    @WithDefault("llama3.1")
    String modelId();
}

and

@ConfigRoot(phase = RUN_TIME)
@ConfigMapping(prefix = "quarkus.langchain4j.ollama")
public interface LangChain4jOllamaConfig {

    /**
     * Default model config.
     */
    @WithParentName
    OllamaConfig defaultConfig();

    /**
     * Named model config.
     */
    @ConfigDocSection
    @ConfigDocMapKey("model-name")
    @WithParentName
    @WithDefaults
    Map<String, OllamaConfig> namedConfig();

    interface OllamaConfig {

// ommitted  config options

        /**
         * Chat model related settings
         */
        ChatModelConfig chatModel();
    }
}

@ConfigGroup
public interface ChatModelConfig {

// ommitted  config options

}

However setting quarkus.langchain4j.ollama.foo.model-id=bar has no effect whatsoever on LangChain4jOllamaFixedRuntimeConfig and I ever see

Unrecognized configuration key "quarkus.langchain4j.ollama.foo.model-id" was provided

in the logs.

Am I doing something wrong or is this not supported?

radcortez commented 2 months ago

Hum , because the path to modelId is :

The configuration name you are trying to set does not contain chat-model. Try quarkus.langchain4j.ollama.foo.chat-model.model-id or add @WithParentName to chatModel()

radcortez commented 2 months ago

BTW, you can omit the double mapping of Map and the Group default with:

@WithParentName
@WithUnnamedKey("default")
@WithDefaults
Map<String, OllamaConfig> namedConfig();

The @WithUnnamedKey allows the Map to accept unnamed keys in the configuration names (as in quarkus.langchain4j.ollama.chat-model.model-id). The default is just the runtime Map key that you need to use to access the values. It can be whatever you want.

We added this to avoid having this double-mapping trick. Feel free to do it as you prefer :)

geoand commented 2 months ago

Ah, I should have seen that myself, sorry to bother you on this :(

I just got mixed with the various levels 🤦🏼

geoand commented 2 months ago

@edeandrea can you confirm that quarkus.langchain4j.ollama.parasol-chat.chat-model.model-id=llama3.1 works for you as expected?

radcortez commented 2 months ago

Ah, I should have seen that myself, sorry to bother you on this :(

No worries, all good. I get mixed up sometimes myself :)

geoand commented 2 months ago

🙏🏼

edeandrea commented 2 months ago

quarkus.langchain4j.ollama.parasol-chat.model-id=llama3.1

Yes I can confirm this works with 0.16.4. I'm still scratching my head as to how I got quarkus.langchain4j.ollama.parasol-chat.model-id. You must have changed something over the weekend and removed all traces of what was there before :) (just kidding).

I don't know what I was thinking. Sorry for the confusion!

geoand commented 2 months ago

Sorry for the confusion!

I'll send the (four digit) bill to your manager 😉

edeandrea commented 2 months ago

Let me know how that works out for you :)

geoand commented 2 months ago

🤑