!extend functionality does not work with substitutions

[jowgn]

The problem

I do use three .yaml files here:

├── licht-arbeitszimmer-iris-waschb.yaml
└── common
    ├── light.yaml
    └── devices
        ├── shelly-1.yaml

licht-arbeitszimmer-iris-waschb.yaml is the obviously main file and uses the other two included as packages. In the light.yaml I need access to the on_state function of the lights input extend it with the trigger to toggle the light. I used to do that with a script that was defined in the light.yaml and executed in the shelly-1.yaml. Of cause !extend would fit my needs better. But !extend only works if I don't use substitutions in it. I saw here that in the Tests is is used with a substitution, so I hope someone can find the problem.

Which version of ESPHome has the issue?

3023.7.1

What type of installation are you using?

Home Assistant Add-on

Which version of Home Assistant has the issue?

No response

What platform are you using?

ESP8266

Board

No response

Component causing the issue

No response

Example YAML snippet

# licht-arbeitszimmer-iris-waschb.yaml
substitutions:
  devicename: licht-arbeitszimmer-iris-waschb
  devicename_h: Licht Arbeitszimmer Iris Waschbecken
  comment: Shelly 1 in der Lampe über Waschbecken im Arbeitszimmer Iris. Verbunden mit einem Leuchtmittel und einem Schalter. (common:${v_common}, device:${v_device})

packages:
  common: !include common/common.yaml
  device: !include 
    file: common/devices/shelly-1.yaml
    vars:
      shelly_output0: "thisRelay"
      shelly_input0: "thisInput"
  light0: !include 
    file: common/light.yaml
    vars:
      light_name: ${devicename_h}
      light_output: "thisRelay"
      light_input: "thisInput"
      light_id: "thisLight"
      light_timer: 4h
# light.yaml
light:
  - platform: binary
    name: ${light_name}
    output: ${light_output}
    id: ${light_id}
    restore_mode: ALWAYS_OFF
    on_turn_on:
      - script.execute: ${light_id}_timer
    on_turn_off:
      - script.stop: ${light_id}_timer

binary_sensor:
  - id: !extend ${light_input}
    on_state:
      script.execute: ${light_id}_on_state

script:
  - id: ${light_id}_timer
    mode: restart
    then:
      - delay: ${light_timer}
      - light.turn_off: ${light_id}
  - id: ${light_id}_on_state
    mode: queued
    then:
      - light.toggle: ${light_id}
# shelly-1.yaml
substitutions:
  v_device: "s1:1.0.0"
  # 1.0.0 copy to use new include

esphome:
  name: ${devicename}
  comment: ${comment}
  platform: ESP8266
  board: esp01_1m

output:
  - platform: gpio
    pin: GPIO4
    id: ${shelly_output0}

binary_sensor:
  - platform: gpio
    internal: true
    pin:
      number: GPIO5
      inverted: true
    # small delay to prevent debouncing
    filters:
      - delayed_on_off: 50ms
    # config for state change of input button
    id: ${shelly_input0}

Anything in the logs that might be useful for us?

No response

Additional information

No response

[sebastianrasor]

I'm reorganizing my configurations and managed to reproduce this. Have you found any workaround?

[sebastianrasor]

I figured out why this issue was never noticed with the tests: https://asciinema.org/a/tKHnVUaMc74qlGRLQKU2he3KA

It seems like !extend is doing some kind of literal string comparison instead of actually resolving the substitution.

[jochocki]

I'm seeing this same issue as well, where !extend is not resolving the substitution. My layout is slightly different, I have the !extend block in a package, and am passing the value in as a variable.

Main yaml file:

substitutions:
  espname: jordan-office-switch
  friendly_name: Jordan Office Switch

packages:
  common: !include packages/common.yaml
  device: !include packages/shelly-plus1.yaml
  lightswitch: !include
    file: packages/lightswitch.yaml
    vars:
      ha_light: light.jordan_office
      relay_id: relay_1
      switch_id: switch_1

lightswitch.yaml file (which contains the !extend):

script:

  - id: ${switch_id}_turn_on
    then:
      if:
        condition:
          api.connected:
        then:
          if:
            condition:
              switch.is_off: ${relay_id}
            then:
              - switch.turn_on: ${relay_id}
              - homeassistant.service:
                  service: light.turn_on
                  data:
                    entity_id: ${ha_light}
            else:
              - homeassistant.service:
                  service: light.turn_on
                  data:
                    entity_id: ${ha_light}
        else:
          - switch.turn_on: ${relay_id}

  - id: ${switch_id}_turn_off
    then:
      if:
        condition:
          api.connected:
        then:
          - homeassistant.service:
              service: light.turn_off
              data:
                entity_id: ${ha_light}
        else:
          - switch.turn_off: ${relay_id}

binary_sensor:
  - id: !extend ${switch_id}
    filters:
      - delayed_on: 10ms
      - delayed_off: 10ms
    on_state:
      if:
        condition:
          binary_sensor.is_on: ${switch_id}
        then:
          - script.execute: ${switch_id}_turn_on
        else:
          - script.execute: ${switch_id}_turn_off

I get the following error when running esphome config against the main yaml file, which shows the variables being substituted correctly, except for the !extend id:

Failed config

binary_sensor.unknown: [source packages/lightswitch.yaml:40]

  Source for extension of ID '${switch_id}' was not found.
  id: !extend ${switch_id}
  filters: 
    - delayed_on: 10ms
    - delayed_off: 10ms
  on_state: 
    if: 
      condition: 
        binary_sensor.is_on: switch_1
      then: 
        - script.execute: switch_1_turn_on
      else: 
        - script.execute: switch_1_turn_off

[kpfleming]

I'm looking into this now; it may be possible, although I wouldn't classify this as a 'bug' since the ! YAML extensions don't allow substitutions in their input data (for example you can't use substitutions to build the pathname for a file to be included using !include).

In spite of that, allowing them to be used in the id-string for !extend could certainly be useful.

[sebastianrasor]

It would definitely be useful for writing more functional configurations, and as written in the tests it seems like it's supposed to work which is why I personally think this is a bug.

Here's an example of a basic configuration which could be greatly improved with this functionality: https://github.com/sebastianrasor/esphome-configs/blob/380e765c2665f40c70a1bb2c67e8576bc8bb31a0/packages/components/on-click.yaml

Here's what that example would look like with this functionality: https://gist.github.com/sebastianrasor/3d8b22c01e5e2b8458aa33630d0c4feb

[kpfleming]

Well.. I've done some work on it, and it's going to be difficult. The cross-checking of ID references happens multiple times during config processing/validation, and most importantly it happens before substitutions are applied. I don't see a practical to delay some (or all) of the ID validation until after the substitutions have been applied.

[shammysha]

Well.. For example, two or more ld2410 on board with tonns of sensors/numbers/etc.. I've make template in esphome for ld2410, where entities defines with id and name, generated by same rules -

1. {devicename}{place} - for name
2. {place}_ - for id This very helpfull for build logic in automation and cards in HA But sometimes I need to use specific filters or or additional logic, for one of LD2410 sensor (placement specifics) And I need to extend entity functionality.. Current problem with !extend functionality limits broke idea of package use :(

[fakuivan]

This would also be useful in my case. What I want to do is add some on_turn_on and on_turn_off actions to perform on an existing switch created in the root config file. I give the include the id of switch and it !extends the config to add those hooks.

[kpfleming]

I've just opened a DRAFT PR which resolves this issue, among others, but does so by providing a different mechanism for extending objects. It does the extension work after packages and substitutions have been processed. If you are interested, take a look at the PR, and in particular the test_extensions.yaml file which has examples of how it would be used.

I have not written documentation for it yet and I won't be surprised if Jesse wants to have the component name be something different from 'extensions', but I've verified that it does work for the existing use cases of !extend and a number of others which are not supported by !extend. If you are willing to speak up and encourage the maintainers to review that PR, please do so, and if it looks like it will make some progress I'll write docs and take it out of draft mode :-)

[sebastianrasor]

A true hero

[Sleeper85]

@kpfleming

How can I use your component in my project?

I receive an error during YAML validation.

It's not possible to import CONF_EXTENSIONS because it does not exist.

from esphome.const import ( CONF_EXTENSIONS, CONF_ID, )

Thanks for your help.

[kpfleming]

That PR's content cannot be used as an external component, because it includes core changes.