search

jekalmin / extended_openai_conversation

Home Assistant custom component of conversation agent. It uses OpenAI to control your devices.
834 stars 108 forks source link

Sorry if this isn't the place but i need help with Extended OpenAI Conversation: Query image #161

Open sneekerzzz opened 4 months ago

sneekerzzz commented 4 months ago

Since i can't find any documentation about this question

I would like to anounce the responds from the Extended OpenAI Conversation: Query image trough my sonos speakers whats the best way to do this? and help in the right direction would be cool

jekalmin commented 4 months ago

Did you want to create a script or register a function of this integration? If you want a script, it should be something like below.

script

tts_image:
  sequence:
  - service: extended_openai_conversation.query_image
    data:
      model: gpt-4-vision-preview
      prompt: "{{prompt}}"
      images:
        url: "{{url}}"
      max_tokens: 300
      config_entry: YOUR_CONFIG_ENTRY_ID
    response_variable: result
  - service: script.google_home_mini_say
    data:
      message: '{{result.choices[0].message.content}}'
  mode: single

Then you can call a script.

service

service: script.tts_image
data:
  prompt: "What's in image?"
  url: "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"