This is custom component of Home Assistant.
Derived from OpenAI Conversation with some new features such as call-service.
Extended OpenAI Conversation uses OpenAI API's feature of function calling to call service of Home Assistant.
Since "gpt-3.5-turbo" model already knows how to call service of Home Assistant in general, you just have to let model know what devices you have by exposing entities
extended_openai_conversation
folder into <config directory>/custom_components
Select "Extended OpenAI Conversation" from "Conversation agent" tab.
After installed, you need to expose entities from "http://{your-home-assistant}/config/voice-assistants/expose".
By clicking a button from Edit Assist, Options can be customized.
Options include OpenAI Conversation options and two new options.
Attach Username
: Pass the active user's name (if applicable) to OpenAI via the message payload. Currently, this only applies to conversations through the UI or REST API.
Maximum Function Calls Per Conversation
: limit the number of function calls in a single conversation.
(Sometimes function is called over and over again, possibly running into infinite loop)
Functions
: A list of mappings of function spec to function.
Edit Assist | Options |
---|---|
native
: built-in function provided by "extended_openai_conversation".
execute_service
domain
(string): domain to be passed to hass.services.async_call
service
(string): service to be passed to hass.services.async_call
service_data
(object): service_data to be passed to hass.services.async_call
.entity_id
(string): target entitydevice_id
(string): target devicearea_id
(string): target areaadd_automation
automation_config
(string): An automation configuration in a yaml formatget_history
entity_ids
(list): a list of entity ids to filterstart_time
(string): defaults to 1 day before the time of the request. It determines the beginning of the periodend_time
(string): the end of the period in URL encoded format (defaults to 1 day)minimal_response
(boolean): only return last_changed and state for states other than the first and last state (defaults to true)no_attributes
(boolean): skip returning attributes from the database (defaults to true)significant_changes_only
(boolean): only return significant state changes (defaults to true)script
: A list of services that will be calledtemplate
: The value to be returned from function.rest
: Getting data from REST API endpoint.scrape
: Scraping information from websitecomposite
: A sequence of functions to execute. Below is a default configuration of functions.
- spec:
name: execute_services
description: Use this function to execute service of devices in Home Assistant.
parameters:
type: object
properties:
list:
type: array
items:
type: object
properties:
domain:
type: string
description: The domain of the service
service:
type: string
description: The service to be called
service_data:
type: object
description: The service data object to indicate what to control.
properties:
entity_id:
type: string
description: The entity_id retrieved from available devices. It must start with domain, followed by dot character.
required:
- entity_id
required:
- domain
- service
- service_data
function:
type: native
name: execute_service
This is an example of configuration of functions.
Copy and paste below yaml configuration into "Functions".
Then you will be able to let OpenAI call your function.
For real world example, see weather.
This is just an example from OpenAI documentation
- spec:
name: get_current_weather
description: Get the current weather in a given location
parameters:
type: object
properties:
location:
type: string
description: The city and state, e.g. San Francisco, CA
unit:
type: string
enum:
- celcius
- farenheit
required:
- location
function:
type: template
value_template: The temperature in {{ location }} is 25 {{unit}}
- spec:
name: add_item_to_shopping_cart
description: Add item to shopping cart
parameters:
type: object
properties:
item:
type: string
description: The item to be added to cart
required:
- item
function:
type: script
sequence:
- service: shopping_list.add_item
data:
name: '{{item}}'
In order to accomplish "send it to Line" like example3, register a notify function like below.
- spec:
name: send_message_to_line
description: Use this function to send message to Line.
parameters:
type: object
properties:
message:
type: string
description: message you want to send
required:
- message
function:
type: script
sequence:
- service: script.notify_all
data:
message: "{{ message }}"
In order to pass result of calling service to OpenAI, set response variable to _function_result
.
- spec:
name: get_events
description: Use this function to get list of calendar events.
parameters:
type: object
properties:
start_date_time:
type: string
description: The start date time in '%Y-%m-%dT%H:%M:%S%z' format
end_date_time:
type: string
description: The end date time in '%Y-%m-%dT%H:%M:%S%z' format
required:
- start_date_time
- end_date_time
function:
type: script
sequence:
- service: calendar.list_events
data:
start_date_time: "{{start_date_time}}"
end_date_time: "{{end_date_time}}"
target:
entity_id: calendar.test
response_variable: _function_result
- spec:
name: play_youtube
description: Use this function to play Youtube.
parameters:
type: object
properties:
video_id:
type: string
description: The video id.
required:
- video_id
function:
type: script
sequence:
- service: webostv.command
data:
entity_id: media_player.{YOUR_WEBOSTV}
command: system.launcher/launch
payload:
id: youtube.leanback.v4
contentId: "{{video_id}}"
- delay:
hours: 0
minutes: 0
seconds: 10
milliseconds: 0
- service: webostv.button
data:
entity_id: media_player.{YOUR_WEBOSTV}
button: ENTER
- spec:
name: play_netflix
description: Use this function to play Netflix.
parameters:
type: object
properties:
video_id:
type: string
description: The video id.
required:
- video_id
function:
type: script
sequence:
- service: webostv.command
data:
entity_id: media_player.{YOUR_WEBOSTV}
command: system.launcher/launch
payload:
id: netflix
contentId: "m=https://www.netflix.com/watch/{{video_id}}"
Before adding automation, I highly recommend set notification on automation_registered_via_extended_openai_conversation
event and create separate "Extended OpenAI Assistant" and "Assistant"
(Automation can be added even if conversation fails because of failure to get response message, not automation)
Create Assistant | Notify on created |
---|---|
Copy and paste below configuration into "Functions"
For English
- spec:
name: add_automation
description: Use this function to add an automation in Home Assistant.
parameters:
type: object
properties:
automation_config:
type: string
description: A configuration for automation in a valid yaml format. Next line character should be \n. Use devices from the list.
required:
- automation_config
function:
type: native
name: add_automation
For Korean
- spec:
name: add_automation
description: Use this function to add an automation in Home Assistant.
parameters:
type: object
properties:
automation_config:
type: string
description: A configuration for automation in a valid yaml format. Next line character should be \\n, not \n. Use devices from the list.
required:
- automation_config
function:
type: native
name: add_automation
Get state history of entities
- spec:
name: get_history
description: Retrieve historical data of specified entities.
parameters:
type: object
properties:
entity_ids:
type: array
items:
type: string
description: The entity id to filter.
start_time:
type: string
description: Start of the history period in "%Y-%m-%dT%H:%M:%S%z".
end_time:
type: string
description: End of the history period in "%Y-%m-%dT%H:%M:%S%z".
required:
- entity_ids
function:
type: composite
sequence:
- type: native
name: get_history
response_variable: history_result
- type: template
value_template: >-
{% set ns = namespace(result = [], list = []) %}
{% for item_list in history_result %}
{% set ns.list = [] %}
{% for item in item_list %}
{% set last_changed = item.last_changed | as_timestamp | timestamp_local if item.last_changed else None %}
{% set new_item = dict(item, last_changed=last_changed) %}
{% set ns.list = ns.list + [new_item] %}
{% endfor %}
{% set ns.result = ns.result + [ns.list] %}
{% endfor %}
{{ ns.result }}
Scrape version from webpage, "https://www.home-assistant.io"
Unlike scrape, "value_template" is added at root level in which scraped data from sensors are passed.
- spec:
name: get_ha_version
description: Use this function to get Home Assistant version
parameters:
type: object
properties:
dummy:
type: string
description: Nothing
function:
type: scrape
resource: https://www.home-assistant.io
value_template: "version: {{version}}, release_date: {{release_date}}"
sensor:
- name: version
select: ".current-version h1"
value_template: '{{ value.split(":")[1] }}'
- name: release_date
select: ".release-date"
value_template: '{{ value.lower() }}'
When using ytube_music_player, after ytube_music_player.search
service is called, result is stored in attribute of sensor.ytube_music_player_extra
entity.
- spec:
name: search_music
description: Use this function to search music
parameters:
type: object
properties:
query:
type: string
description: The query
required:
- query
function:
type: composite
sequence:
- type: script
sequence:
- service: ytube_music_player.search
data:
entity_id: media_player.ytube_music_player
query: "{{ query }}"
- type: template
value_template: >-
media_content_type,media_content_id,title
{% for media in state_attr('sensor.ytube_music_player_extra', 'search') -%}
{{media.type}},{{media.id}},{{media.title}}
{% endfor%}
Question: When did bedroom light turn on?
Query(generated by gpt-3.5): SELECT * FROM states WHERE entity_id = 'input_boolean.livingroom_light_2' AND state = 'on' ORDER BY last_changed DESC LIMIT 1
- spec:
name: query_histories_from_db
description: >-
Use this function to query histories from Home Assistant SQLite database.
Example:
Question: When did bedroom light turn on?
Answer: SELECT datetime(s.last_updated_ts, 'unixepoch', 'localtime') last_updated_ts FROM states s INNER JOIN states_meta sm ON s.metadata_id = sm.metadata_id INNER JOIN states old ON s.old_state_id = old.state_id WHERE sm.entity_id = 'light.bedroom' AND s.state = 'on' AND s.state != old.state ORDER BY s.last_updated_ts DESC LIMIT 1
Question: Was livingroom light on at 9 am?
Answer: SELECT datetime(s.last_updated_ts, 'unixepoch', 'localtime') last_updated, s.state FROM states s INNER JOIN states_meta sm ON s.metadata_id = sm.metadata_id INNER JOIN states old ON s.old_state_id = old.state_id WHERE sm.entity_id = 'switch.livingroom' AND s.state != old.state AND datetime(s.last_updated_ts, 'unixepoch', 'localtime') < '2023-11-17 08:00:00' ORDER BY s.last_updated_ts DESC LIMIT 1
parameters:
type: object
properties:
query:
type: string
description: A fully formed SQL query.
function:
type: sqlite
Get last changed date time of state | Get state at specific time |
---|---|
FAQ
No, since connection is created in a read only mode, data are only used for fetching.
Yes, it is hard to validate whether a query is only using exposed entities.
Yes. Set "TZ" environment variable to your region (eg.
Asia/Seoul
).
Or use plus/minus hours to adjust instead of 'localtime' (eg.datetime(s.last_updated_ts, 'unixepoch', '+9 hours')
).
See more practical examples.
In order to monitor logs of API requests and responses, add following config to configuration.yaml
file
logger:
logs:
custom_components.extended_openai_conversation: info