[#43] add "query_image" service

jekalmin commented 11 months ago

Objective

Add "query_image" service to upload an image to gpt
Although it's not ideal, let gpt know via function response content returned from service call (extended_openai_conversation.query_image).

Service

Example

I tested with following image searched from website

Function

- spec:
    name: get_refrigerator_items
    description: Get description of items in refrigerator
    parameters:
      type: object
      properties:
        url:
          type: string
          description: image url of refrigerator
          enum:
            - https://i.pinimg.com/originals/8b/cc/f1/8bccf14daf77ce887fc162934335cb21.jpg # needs to change
      required:
      - url
  function:
    type: composite
    sequence:
      - type: script
        sequence:
          - service: extended_openai_conversation.query_image
            data:
              prompt: What alcohol and brands do you see in this picture?
              images:
                - url: "{{url}}"
              max_tokens: 300
              config_entry: YOUR_CONFIG_ENTRY_KEY # needs to change
            response_variable: _function_result
        response_variable: image_result
      - type: template
        value_template: "{{image_result.choices[0].message.content}}"

TBTHEDBA commented 10 months ago

How did I add this service?

jekalmin commented 10 months ago

How did I add this service?

I haven't released it yet. Let me release this in beta.

jekalmin commented 10 months ago

@TBTHEDBA I just released this in 1.0.1-beta2. Please try this and give a feedback :D

TBTHEDBA commented 10 months ago

I received this error in debugging. Should I be using this as my "your_config_entry_key". I got it from hacs.repositories "700780425": { "repository_manifest": {

2024-01-17 09:53:53.608 ERROR (MainThread) [custom_components.extended_openai_conversation.helpers] extended_openai_conversation: Error executing script. Invalid data for call_service at pos 1: expected str for dictionary value @ data['config_entry'] 2024-01-17 09:53:53.609 ERROR (MainThread) [homeassistant.components.assist_pipeline.pipeline] Unexpected error during intent recognition Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/components/assist_pipeline/pipeline.py", line 988, in recognize_intent conversation_result = await conversation.async_converse( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

On Wed, Jan 17, 2024 at 8:22 AM Min Jekal @.***> wrote:

@TBTHEDBA https://github.com/TBTHEDBA I just released this in 1.0.1-beta2. Please try this and give a feedback :D

— Reply to this email directly, view it on GitHub https://github.com/jekalmin/extended_openai_conversation/pull/60#issuecomment-1895915046, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJXIGPX3CHV3TUJO45QCUOTYO7NBHAVCNFSM6AAAAABBBT4U4OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOJVHEYTKMBUGY . You are receiving this because you were mentioned.Message ID: @.***>

jekalmin commented 10 months ago

Try to call service from developers tools. You can get entry key from there.

ui mode	yaml mode

TBTHEDBA commented 10 months ago

[image: image.png] That did it! Thank you so much for your help. Now I get to shame my children for not cleaning their room.

On Wed, Jan 17, 2024 at 10:28 AM Min Jekal @.***> wrote:

Try to call service from developers tools. You can get entry key from there.

Screenshot_20240118_012737_Home.Assistant.jpg (view on web) https://github.com/jekalmin/extended_openai_conversation/assets/2917984/bc5de19d-44b9-42be-9411-33d550436bcd

— Reply to this email directly, view it on GitHub https://github.com/jekalmin/extended_openai_conversation/pull/60#issuecomment-1896165429, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJXIGPUGWDWFYKZAVOFR4J3YO733HAVCNFSM6AAAAABBBT4U4OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOJWGE3DKNBSHE . You are receiving this because you were mentioned.Message ID: @.***>

jekalmin / extended_openai_conversation