search

zmedelis / bosquet

Tooling to build LLM applications: prompt templating and composition, agents, LLM memory, and other instruments for builders of AI applications.
https://zmedelis.github.io/bosquet/
Eclipse Public License 1.0
280 stars 19 forks source link

exceptions in llm-generate are invisible #21

Closed behrica closed 1 year ago

behrica commented 1 year ago

bosquet does not show them in any form, I assume that Selmers just renders "empty" string when a tag fails

behrica commented 1 year ago

the exceptins are only swalloed in gen/complete.

In use_guide.clj:

(def review (gen/complete play-review
                          {:title "Mr. X" :genre "crime"}
                          [:synopsis :evrything]
                          {}
                          ;{:synopsis { :llm-generate open-ai-config}
                          ; :evrything {:llm-generate azure-open-ai-config}}
                          ))

-> should fail as model key is not given

But it continues instead and gives empty result:

review:
{:title "Mr. X", :genre "crime", :synopsis nil, :play nil, :evrything nil, :review nil}

gen/complete-template works and exception is thrown

zmedelis commented 1 year ago

Fixing this with https://github.com/zmedelis/bosquet/commit/74b1d8e6a84ff89a5f6f940b90258106bbd9f3eb It uses Pathom error handling plugin https://pathom3.wsscode.com/docs/error-handling/#observing-resolver-errors

The behavior remains the same - nil is returned for the field that resulted in an error, but the error will be logged.

I was thinking of exposing the registration of different error handlers (maybe one wants to retry a call to LLM). Decided against it at this point. It complicates Bosquet API without knowing the clear benefits of doing that. More generally the whole Pathom env construction could be more open for the client, allowing way more configuration in processing.