Compiles are not independent

[0x0a11c0de]

The problem

When you create reusable components, esphome does compiles that are not independent from each other. This creates two issues:

1. The build cache is invalidated for other compiles.
2. Parallel compiles cannot be done.

For example, I create a reusable yaml file and then use substitutions to create different devices:

esphome -s human_name "Living Room Switch" -s device_name "living-room-switch" compile "common/treatlife_ss01s.yaml"
esphome -s human_name "Porch Light Switch" -s device_name "porch-light-switch" compile "common/treatlife_ss01s.yaml"

When these are compiled, it creates a directory structure like this:

common/
    .esphome/
        build/
            living-room-switch/
            porch-light-switch/
        idedata/
            living-room-switch/
            porch-light-switch/
        treatlife_ss01s.yaml.json

Everything is independent except for treatlife_ss01s.yaml.json. It gets changed every time. This means after I do the porch-light-switch compile, the living-room-switch compile will be redone from scratch again even if I've already compiled it.

Which version of ESPHome has the issue?

2021.12.3

What type of installation are you using?

pip

Which version of Home Assistant has the issue?

No response

What platform are you using?

ESP8266

Board

No response

Component causing the issue

No response

Example YAML snippet

substitutions:
  human_name: "Treatlife SS01S"
  device_name: treatlife-ss01s
  icon: "mdi:light-switch"
  led_inverted: "True"

esphome:
  name: $device_name
  platform: ESP8266
  board: esp01_1m

packages:
  common: !include common.yaml

logger:

binary_sensor:
  - platform: gpio
    pin:
      number: GPIO4
      inverted: $led_inverted
    id: power_sense
    internal: True

  - platform: gpio
    pin:
      number: GPIO13
      inverted: True
    id: button
    name: ${human_name} Button
    on_press:
      - switch.toggle: switch_template
    internal: True

light:
  - platform: binary
    id: white_led
    output: led_1
    restore_mode: RESTORE_DEFAULT_ON
    internal: True

output:
  - platform: gpio
    id: led_1
    pin:
      number: GPIO4
      inverted: $led_inverted

switch:
  - platform: gpio
    id: relay
    pin:
      number: GPIO12
    restore_mode: RESTORE_DEFAULT_OFF
    internal: True

  - platform: template
    name: ${human_name}
    id: switch_template
    icon: ${icon}
    lambda: |-
      if (id(power_sense).state) {
        return false;
      } else {
        return true;
      }
    turn_on_action:
      - if:
          condition:
          - binary_sensor.is_on: power_sense
          then:
          - switch.turn_on: relay
          - light.turn_off: white_led
    turn_off_action:
      - if:
          condition:
          - binary_sensor.is_off: power_sense
          then:
          - switch.turn_off: relay
          - light.turn_on: white_led

Anything in the logs that might be useful for us?

No response

Additional information

No response

[oxan]

The underlying problem here is that substitutions aren't supported in the device name.

[fhriley]

Other than what I noted above, device_name substitution has been working fine for me. If it’s not officially supported, IMO it should be. Having a single yaml for a type of device (e.g. a light switch) that is deployed with some substitutions works great when you have a lot of them.

[OttoWinter]

The underlying problem here is that substitutions aren't supported in the device name.

Hm I think that's not the issue per se.

The issue is: ESPHome expects there to be one file per phyiscal device. So one living-room-switch.yaml file and one porch-light-switch file. This is so that the .esphome/....json entries are by the .yaml filename, and thus can be resolved by the dashboard without having to load the YAML file.

So the layout you should be using is something like this

one porch-light-switch.yaml file, with some substitutions set at in the file and extending the treatlife_ss01s.yaml file using packages.

[0x0a11c0de]

IMO, this should be fixed. It prevents true reusability. Why should I have to create 20 lightswitch yaml files when the only difference between them is the device name? For example, my current SOP when adding a new treatlife switch is 1) add a new DNS entry and 2) esphome -s.. treatlife_ss01s.yaml. That's it. I don't have to add yet another yaml file, and it's guaranteed to have the same config as all of my other light switches.

[jesserockz]

As per above, this is all that is needed for one file per device and they will have the exact same config except the names. Basically the equivalent of you having one command with substitutions per device.

substitution:
  human_name: Porch
  device_name: porch

packages:
  common: !include treatlife_ss01s.yaml

[0x0a11c0de]

That's the problem. I'd have to create and maintain 20+ files just for switches, and all those files are doing is setting device names. How would a reasonable person (or at least, any developer who is looking for maximal reuse) not think that the -s option is the correct solution here? The answers I'm getting seem to indicate that -s is inconsistent in when it is expected to work.

[OttoWinter]

That's the problem. I'd have to create and maintain 20+ files just for switches, and all those files are doing is setting device names.

Aren't you already doing just that by having to type in the name each time you run the command?

As @jesserockz mentioned, these files would be really simple (just 5 lines), and basically just do what you were already doing from the command line.

not think that the -s option is the correct solution here?

So the -s was really just added as a convenience thing for some setups where you want to quickly switch out something without editing the file (I think, but I don't think I was around when it was added). The way it's set up here it can't work for substituting device_name.

We optimize the user experience for the dashboard view. And there it's important to have real-time information about devices (which is only possible by indexing the .esphome entries by YAML filename, as we can't load the full .yaml file all the time - especially when it's got temporary syntax issues). And tbh I'm contemplating just dropping the -s option as as you said it can really be quite inconsistent in how it's meant to work.

[fhriley]

That's the problem. I'd have to create and maintain 20+ files just for switches, and all those files are doing is setting device names.

Aren't you already doing just that by having to type in the name each time you run the command?

No. I add a target to a single file: a makefile. Altogether I have 7 yaml files and one makefile that covers 50+ devices. Obviously this is much easier to maintain than 50+ files.

As @jesserockz mentioned, these files would be really simple (just 5 lines), and basically just do what you were already doing from the command line.

So I'd have to add 20+ files for switches, another 20+ for plugs.

So the -s was really just added as a convenience thing for some setups where you want to quickly switch out something without editing the file (I think, but I don't think I was around when it was added).

That's what I'm using it for?

The way it's set up here it can't work for substituting device_name.

But it does work. The only issues are what I noted in the OP.

We optimize the user experience for the dashboard view. And there it's important to have real-time information about devices (which is only possible by indexing the .esphome entries by YAML filename, as we can't load the full .yaml file all the time - especially when it's got temporary syntax issues). And tbh I'm contemplating just dropping the -s option as as you said it can really be quite inconsistent in how it's meant to work.

That's a shame. You're missing an opportunity for esphome to be the solution for truly reusable components. Users could create sensor devices without having to write any yaml. If I'm not wasting my time, I could give an example of what I mean.