Adding some programmability capabilities to yaml configs

Describe the problem you have/What new integration you would like

It would be nice to extend yaml configuration with some programmability features which would allow modifying configuration without direct changes to the yaml code defining components. Instead it will be achievable by setting configuration options.

By programmability I mean things like:

conditions - e.g. include or exclude platform component from the config based on the value of substitution.
loops - e.g. create multiple similar components basing on the config defined once.
expressions - e.g. extend substitutions capabilities to support expressions. This will allow us to inject information available in build time and will extend templating capabilities.

The proposed solution will be to add a term operator which is essentially a key in component definition prefixed with some character e.g. _. This key is transparent for component configuration but might modify config by adding, excluding, or transforming existing definitions. See the examples below.

Please describe your use case for this integration and alternatives you've tried:

Scenario 1. Different hardware versions of the same device require slightly different configuration (conditions use case).

Let's say we have a device that could exist in a few variations. To be more specific let it be some universal air quality sensor designed in the way when PCB has slots for 5 different sensors. On the assembly stage, you decide which sensors to use.

The problem: For each combination of sensors you have to create a separate config file. So a) you have to manage a bunch of very similar files b) you will need some naming convention to give reasonable names

Proposed solution:

Simplistic approach (if we don't have expressions):

substitutions:
  disable_bme280: false
  disable_mhz19: true
sensors:
  - platform: mhz19
    _exclude: $disable_mhz19
    co2:
      name: "CO2"
    temperature:
    name: "Temperature"
    update_interval: 60s
  - platform: bme280
    _exclude: $disable_bme280
    temperature:
      name: "BME280 Temperature"
      id: bme280_temperature
    pressure:
      name: "BME280 Pressure"
      id: bme280_pressure
    address: 0x77
    update_interval: 15s

In this case by changing substitution value OR which is more likely by passing command line argument we could include\exclude sensors. This solution is based on _exclude operator which excludes a piece of configuration if the value is true.

The other and probably a more intuitive option would be _if operator which does the same but expects the opposite value.

Scenario 2. One device might require the bunch of similar set's of components (loops use case)

Consider the following example. We have a PCB for a device that controls floor heating in the house. It could manage up to 6 zones. Each zone needs a temperature sensor + relay to control the valve. The PCB is universal so it has a place for components for 6 zones, but again you could solder as just 4 if you don't need more.

The Problem: a) You have to copy and paste a lot of code. For each zone you will need at least sensor, output, switch, bangbang controller. b) when you need multiple devices like this, again you have to manage a bunch of very similar files c) regardless of home many physical sensors\relays you have on board if in fact, you need just 2 of 6 available you don't want to home assistannt to show and track excessive entities that are not in use.

Proposed Solution

We have _count operator which duplicates config as many times as we need. It also exposes the special variable loop.counter` which will be available inside the definition.

See example:

substitutions:
  zones_count: 3
  not_expose_valves: true
sensor:
  - platform: dallas
    _count: $zones_count
    address: 0x1c0000031edd2a28
    name: "Zone ${loop.counter} temperature"
    id: "temperature_zone_${loop.counter}"
outputs:
  - platform: gpio
    count: $zones_count
    id: zone${loop.counter}_output
    pin:
      number: ${loop.counter}
      pcf8574: io_hub0
switch
  - platform: output
    internal: "${not_expose_valves}"
    name: "Zone ${loop.counter} valve"
    id: "_valve${loop.counter}"
    output: ${loop.counter}_output
climate:
  - platform: bang_bang
    id: "zone${loop.counter}_controller"
    name: "Zone ${loop.counter}"
    sensor: "zone${zone1_id}_temperature"
    heat_action:
      - switch.turn_on: "zone${loop.counter}_valve"
    idle_action:
      - switch.turn_off: "zone${loop.counter}_valve"

Scenario 3. Adding information available in build time into config (use case for expressions)

It might be very useful to expose information about config which is currently running especially when you have a lot of devices and have a sort of CI.

Proposed Solution: Allow to evaluate some template language expressions e.g. jinja2 which might replace substitutions in future and will give much more flexibility.

text_sensor:
  - platform: template
    name: "Firmware Build Date"
    lambda: |-
      return {"{{ date(now)|format(yyyymmdd) }}"};
  - platform: template
    name: "Firmware Config"
    lambda: |-
      return {"{{ runtime.config_txt|sha256 }}"};

Also, this will resolve FR #804.

Scenario 4. More flexibility with conditions and loops (use case for expressions)

To make it defining loops and conditions more comfortable simple substitutions will be not enough. There are multiple examples where you might need expressions. Just a few ones:

Simple math to defined correct values in the loops. Let's say for example below numbering of pins pcf8574 starts not from 0:
```
- platform: gpio
count: $zones_count
id: zone${loop.counter}_output
pin:
  number: ${loop.counter + 2}
  pcf8574: io_hub0
```

Using text functions to follow naming convention and do not repeat yourself:

- platform: gpio
count: $zones_count
id: {{zones[loop.counter].name|camelcase}}
name: {{zones[loop.counter].name|capitalize}}

Using more complex conditions:

- platform: gpio
_if: {{ myVar == '123' and anotherVar == '456' }}
id: {{zones[loop.counter].name|camelcase}}
name: {{zones[loop.counter].name|capitalize}}

Scenario 5. Going a little bit further with Packages feature

While initially packages were built mainly to help better organize the code implementing programmability features might become a good background to create universal, sharable, reusable configurations and a kind of open storage with ready to use packages. Basically the similar to Ansible Galaxy or Terraform Registry

Additional context

Now I manage around 15 devices running ESPHome, so the problem I'm raising will be relevant rather for ppl who reached the state when it's time to think about build and deployment automation to keep things organized then for the average hobbyist.

The proposed solution was inspired by the approach implemented in Terraform, a tool I frequently use to define cloud infrastructure as a code and manage infrastructure changes, deployments, and other staff. Basically they met pretty similar issue some time ago and the solution proved its viability.

Some other considerations

For simplicity proposes in my examples I used substitutions but in fact in the way they are implemented right now, it won't work. We will have to either re-implement the way they are processed and treated (which is a risk in terms of backward compatibility) or implement separate the mechanism which will be processed in another way.

The goal of this long read is to ask the community for feedback on the concept. Right now it is rather in the status of idea and a lot of details need to be clarified and designed. I'm ready to help with implementation (while it will not be fast) but before I move on I'd like to see if this finds any support.

Yup. This would be rad.

I agree such a feature would be quite nice and especially help with complex setups.

Getting this done right however is very hard, especially since any solution that is created essentially will be permanent (changing the system afterwards in significant ways would not be easy since especially vocal groups would use it).

I've played with ansible a bit over the last ~year ish, and I really like the templating system. Using jinja2 after parsing the YAML instead of before parsing YAML is a very good choice and ESPHome should use that too.

Some other notes:

It should be opt-in, configs that accidentally use jinja2 syntax (can happen especially with homeassistant.service action) need to be marked with a tag like !lambda.
The system should be clearly separated from the runtime templating system !lambda. A big fear of mine is that this will become a major point of confusion for beginners.
Using "magic keys" like _exclude in your example is not so nice IMO, because integrations might want to use the keys themselves. However, I don't really know an alternative to this. Definitely all such keys would need to have a common prefix.

I'd like this effort to be driven by example configurations. It would be great if we could gather some example complex configurations used in the wild. Then we can argue better about what would work / what not.

One major thing that would solve ~80% of problems (I think) would already just be an !include directive where you can specify custom substitutions. Like say !include dht22.yaml with data_pin=GPIO12 (obviously not in that syntax). That would also be the easiest thing to implement of all scenarious described above, but it has to fit into the rest of the system once that comes.

Hey @OttoWinter great to hear you see some value here. I like the idea to make it example driven - I'll try to come up with some initial cases next week.

A few immediate comments to your message:

Using "magic keys" like _exclude in your example is not so nice IMO, because integrations might want to use the keys themselves. However, I don't really know an alternative to this. Definitely all such keys would need to have a common prefix.`

Agree. At the same time I also don't think we will be able to figure out any better approach. If we will look around to see how other projects do this there is always something similar. E.g. Ansible has a list of reserved words which define behavior, not configuration option (for example with_items syntax). Terraform does it in the similar way e.g. count syntax. The bad thing I don't like is the lack of clear designation of meta-properties and regular properties. You have to always consult with docs to figure out if this property is a part of component configuration or not. This is especially annoying when you deal with modules written by someone else.

One major thing that would solve ~80% of problems (I think) would already just be an !include directive where you can specify custom substitutions. Like say !include dht22.yaml with data_pin=GPIO12 (obviously not in that syntax). That would also be the easiest thing to implement of all scenarious described above, but it has to fit into the rest of the system once that comes.

I'd rather bet for further improving packages concept. One thing I really love about ESPHome is strict config schema validation. So when I worked on packages I though that the next step if\when we implement some programmability would be to convert it into reusable self-containing modules with strictly defined interface. Consider example below:

Package file

package:
  name: Air sensors
  author: Name <some@email.com>
  params:
    i2c_bus:
      required: true
      type: id
    dht22_pin: 
      required: false
      type: pin
      default: GPIO1
    name_prefix: 
      required: false
      type: string
      default: Sensor 

sensor:
  - platform: dht
    pin: !expression {{pkg_params.dht22_pin}}
    temperature:
      name: "!expression {{pkg_params.name_prefix}} Temperature"
    humidity:
      name: "!expression {{pkg_params.name_prefix}} Humidity"
      update_interval: 60s
.......

In main configuration

esphome:
.....
packages:
  sensors:
    source: ./packages/sensors.pkg.yaml
    params:
       dht22_pin: GPIO21
       name_prefix: Living room

A few months ago I actually wrote a Python package that kind of does exactly this, in the form of a preprocessor.

My tool needs two files, a YAML template (the actual EspHome YAML file, but with conditionnal defines and placeholders) and a YAML device file (that defines the devices, with defines and values). It also supports device inheritance.

With this tool, a template looks like this:

esphome:
  name: esp_{#id}
  platform: {#platform}
  board: {#board}
  #rem We must use an IFDEF instead of a DROP because esphome doesn't allow empty on_boot section
  #ifdef on_boot
  on_boot:
    {#on_boot}
  #endif

logger:
  level: {#logger_level or default INFO}

#ifndef disable_status_led
status_led:
  pin:
    number: LED
    #ifdef invert_status_led
    inverted: True
    #endif
#endif

#ifdef wifi_ssid
wifi:
  ssid: {#wifi_ssid}
  password: {#wifi_password or drop}
  #ifndef disable_wifi_fast_connect
  fast_connect: true
  #endif
  #ifdef static_ip
  manual_ip:
    static_ip: {#static_ip}
    gateway: {#static_ip_gateway or error Static IP defined without a gateway}
    subnet: {#static_ip_subnet or default 255.255.255.0}
    dns1: {#static_ip_dns1 or default 1.1.1.1}
    dns2: {#static_ip_dns2 or default 1.0.0.1}
  use_address: {#use_address or drop}
  #else
    #info No static IP defined, DHCP will be used
  #endif

#ifndef disable_mqtt
mqtt:
  broker: {#mqtt_broker}
  username: {#mqtt_username}
  password: {#mqtt_password}
  topic_prefix: $device_topic
  discovery: false
#endif

#ifdef i2c
i2c:
  sda: {#i2c_sda_pin or drop}
  scl: {#i2c_scl_pin or drop}
  #ifdef i2c_scan
  scan: True
  #endif
#endif

light:
  {#lights or drop}

(more stuff here)

And de device file looks like this:

inherits:
  base.yaml:
defines:
  platform: ESP8266
  board: d1_mini
  invert_status_led:
  static_ip: 192.168.0.209
  floor: upstairs
  room: entree
  uart:
  uart_tx_pin: D0
  uart_rx_pin: D1
  uart_baud_rate: 9600
  dfplayer:
  binary_sensors:
    - platform: gpio
      id: doorbell_button
      pin:
        number: D6
        mode: INPUT_PULLUP
      name: "Doorbell button"
      id: "doorbell_button"
      filters:
        - invert:
        - delayed_on: 100ms
      on_press:
        then:
          - dfplayer.set_volume: 30
          - dfplayer.play: 1
          - mqtt.publish:
              topic: home/doorbell
              payload: rang

(the base.yaml file is the parent device, which defines all common stuff like WIFI name/pwd, mwtt, etc.)

I've been using it excluvely since I coded it, helped me avoid a lot of repetition, especially in cases where I have multiple identical devices (one temp sensor per room, one leak detector per monitored location).

Sadly, the tool is not documented at all, but if anybody is interested I'd gladly give some basic info and if still interested I could put up an official documentation.

esphome / feature-requests