search

zmedelis / bosquet

Tooling to build LLM applications: prompt templating and composition, agents, LLM memory, and other instruments for builders of AI applications.
https://zmedelis.github.io/bosquet/
Eclipse Public License 1.0
280 stars 19 forks source link

Use OpenAI vision #56

Open AvivSpiritt opened 2 months ago

AvivSpiritt commented 2 months ago

Hi,

I'm trying to use OpenAI's vision capabilities, but I'm unsure how to pass an image using image_url. When I tried using it, it seems the model treats image_url like any other text. (not sure if I can pass an object instead of text)

  (generate
   [[:system "You help identify images"]
    [:user [{:type "text"
             :text "Whats in this image?"}
            {:type "image_url"
             :image_url {:url base64-image}}]]
    [:assistant (llm :openai-azure)]])

Thanks 🙏