Template: automatic analysis does NOT find all entities that affect the state in the template

[Molodax]

The problem

The template does not find all entities that affect the state in the template. In the example below, only 3 entities were identified by the automatic analysis:

• binary_sensor.aeon_sensor
• light.dimmer
• sensor.motions_in_15_min

While there are two more entities that affect the state in the template:

• binary_sensor.motion_in_room
• group.all_persons

Environment

• Home Assistant Core release with the issue: 0.115.x
• Last working Home Assistant Core release (if known): 0.114.x
• Operating environment (OS/Container/Supervised/Core): Supervised
• Integration causing this issue: template
• Link to integration documentation on our website: https://www.home-assistant.io/integrations/binary_sensor.template/

Problem-relevant configuration.yaml

'{{as_timestamp(now()) - as_timestamp(states.binary_sensor.aeon_sensor.last_changed|default(now())) < 930 or is_state("binary_sensor.aeon_sensor", "on") or (states.light.dimmer.state == "on" and states.sensor.motions_in_15_min.state|int < 1 and (states.binary_sensor.aeon_sensor.last_changed.timestamp()- states.binary_sensor.motion_in_room.last_changed.timestamp()) > 60 and states.group.all_persons.state == "home" ) }}'

Traceback/Error logs

Additional information

[probot-home-assistant[bot]]

Hey there @phracturedblue, @tetienne, mind taking a look at this issue as its been labeled with an integration (template) you are listed as a codeowner for? Thanks! _{^{(message by CodeOwnersMention)}}

[tdejneka]

I tested your template on my system (0.115.3) and the Template Editor reported only one entity was assigned a listener. I modified the template slightly, to make the entities even easier to identify, and it still only assigned one listener.

I removed parts of the template and then it did better but still missed the group.

Most unusual was this variation where it reported it would listen to all entities:

---

EDIT

Clarification: none of these entities exist on my system. However, I don't think that affects Home Assistant's ability to identify entities in a template.

[KevinCathcart]

I think this may be functioning mostly as intended (despite seeming confusing). My understanding is that the set of entities listened for is now dynamic, based on the set of entities actually evaluated during the last template render.

For the original's poster's example: Because the first two cases evaluated to false, it starts evaluating the "and" cases. The first clause states("light.dimmer") == "on" evaluated to true, so it moved onto states.sensor.motions_in_15_min.state|int < 1 which was false and short circuits the rest. So the remaining entities don't yet get a listener attached, but it does not matter, because they cannot make any difference unless and until states.sensor.motions_in_15_min's state becomes 1 or greater. If that happens, then it will re-evaluate, and will end up adding listeners for more entities.

Following this sort of logic, the first two screenshots also make sense, given that none of the entities exist on that system.

However, that "all states changed events" one though looks really unfortunate, and undesirable. Trying to evaluate some non-existent entity's state probably should not be able to escalate to an all states listener, but it looks like that is what happened?

[Molodax]

I think this may be functioning mostly as intended (despite seeming confusing). My understanding is that the set of entities listened for is now dynamic, based on the set of entities actually evaluated during the last template render.

In fact, if it is true, it undermines the whole idea of a template. It is wrong thinking to implement logic in steps since it is not an automation. There is an expectation that the template expression should give a correct true or false at given point of time and not under condition whether some parts of the template were true or false time ago.

[bdraco]

@KevinCathcart is correct. There is no reason to listen to any more entities once you reach a condition that will always force the rest of the template to not evaluate since that entity needs to change state before the template evaluation can ever continue. If we listened for the other entities, we would keep reprocessing the template needlessly because they can never affect the state of the template until the entity that is short-circuiting the template changes state.

@tdejneka Your last variation is likely generating an exception UndefinedError: 'None' has no attribute 'state' so it has to listen for everything.

[Molodax]

If we listened for the other entities, we would keep reprocessing the template needlessly because they can never affect the state of the template until the entity that is short-circuiting the template changes state.

This is not correct, in the example above you have entities that affect the template, though it will never be accounted with the current implementation from 0.115.

[bdraco]

As soon as binary_sensor.aeon_sensor changes state to on or the last changed condition is true, the listener will be updated.

[Molodax]

As soon as binary_sensor.aeon_sensor changes state to on or the last changed condition is true, the listener will be updated.

This is the point that if binary_sensor.aeon_sensor remains unchanged but e.g. group.all_persons becomes not_home, the template will not change to false anymore.

[Molodax]

There is another example where the behaviour will be different not due to a different logic but simply because expression in brackets comes either in the beginning or in the end which makes templates ridiculously odd now:

'{{ (states.input_boolean.weekdays.state==states.binary_sensor.workday_sensor.state=="on" or states.input_boolean.weekend.state!=states.binary_sensor.workday_sensor.state=="off") and states.input_datetime.timerswitch_on.attributes.timestamp | int | timestamp_custom("%H:%M", False) == states.sensor.time.state and states.switch.telldus_tzwp_102_plug_in_switch_switch.state == "off" }}'

'{{ states.input_datetime.timerswitch_on.attributes.timestamp | int | timestamp_custom("%H:%M", False) == states.sensor.time.state and states.switch.telldus_tzwp_102_plug_in_switch_switch.state == "off" and (states.input_boolean.weekdays.state==states.binary_sensor.workday_sensor.state=="on" or states.input_boolean.weekend.state!=states.binary_sensor.workday_sensor.state=="off") }}'

[bdraco]

jinja doesn't document precedence rules, however the below may be helpful:

https://stackoverflow.com/questions/23248850/order-of-operations-for-jinja2-filters https://community.home-assistant.io/t/be-mindful-of-operator-and-filter-precedence-in-template-expressions/99835

[Molodax]

jinja doesn't document precedence rules, however the below may be helpful:

https://stackoverflow.com/questions/23248850/order-of-operations-for-jinja2-filters https://community.home-assistant.io/t/be-mindful-of-operator-and-filter-precedence-in-template-expressions/99835

Thanks, interesting. However, it stil doesn't resolve the issue introduced in 0.115. It, actually, makes this even more the issue than before.

[KevinCathcart]

The basic idea is that if you evaluate something, and can reach a conclusion while only evaluating certain parts, then the fact that other parts you did not even need to consider have changed cannot change your overall conclusion, at least until one of the criteria you did check has changes. As soon as a condition that was considered in reaching the conclusion changes, it must be re-evaluated, and other conditions might start becoming relevant.

Perhaps it would help to give some examples:

Let's say my condition was `the dungeon door is open and there is a unicorn on the 3rd floor balcony'. If I evaluate this, and I determine the dungeon door is closed, I can conclude my condition is false, without even checking the balcony.

If I am interested in knowing when this condition changes, and am watching the dungeon door, then right now I don't need to have somebody watch the balcony. Even if a unicorn show up there, it cannot make my condition true while the light switch is still off, since both parts need to be true, and one is not. As soon as I see the door open, I do need to check for and start keeping track of magical equines on the balcony, but not before then.

Similarly this can work with "or" conditions. If I have a condition of "the ogres are hungry or the Queen is visiting" and I determine the ogres are in fact hungry, I don't need to check if Her Majesty is here in order to know that the condition is true. Only once the ogres have eaten (or otherwise ceased being hungry), do I need to check/keep track of what royalty is visiting. (Well, it is probably not a bad idea to keep track of that anyway, but is it not required to know if my condition is true :smile:).

These same ideas are simply being applied here.

[Molodax]

The basic idea is that if you evaluate something, and can reach a conclusion while only evaluating certain parts, then the fact that other parts you did not even need to consider have changed cannot change your overall conclusion, at least until one of the criteria you did check has changes.

There is a fundamental flaw in such thinking because it implies that events you are evaluating are dependent and, therefore, you perform your evaluation in steps. However, it is not always a true since the issue has been raised not for automation template condition only but also e.g. for binary occupancy sensor. Try your "funny" examples in slightly complicated and more real scenario:

Imagine that you have to learn `the dungeon door is open and there is a unicorn on the 3rd floor balcony' to know whether your unicorn is still in dungeon or already escaped to the 3rd floor. Your door sensor cannot report whether the door is open or closed, it can only report whether the door was moving (a simple motion sensor with some latency period). Simple change of the door sensor from off to on and after some latency to off state doesn't allow you to say whether your unicorn managed to escape or not, that's why you need to know if it appears on the 3rd floor. The new implementation of templates analysis makes it even "funnier" because now your template will behave differently depending if you check the balcony or the door first and their states.

This makes templates "useful" in a very narrow application and not a logical/reliable/universal tool anymore. Again, it worked prior to 0.115.

[KevinCathcart]

I made no assumption that the events are dependent. I never assumed the unicorn was in the dungeon. These are two completely independent events. Nor are we evaluating the condition in steps. We always evaluate the condition from scratch. What is done is short circuiting. If while evaluating a condtion it becomes possible to reach a conclusion on the whole condition without evaluating part of it then we stop, because there is no point in evaluating the rest if it cannot possibly change the outcome. (Just like in a best 2 out of 3 match of some game, people usually don't bother to play the third game if the same person won the first two, since the third one cannot change the overall outcome).

Consider a more obviously independent scenario: two old school light switches. Lets assume they are not even connected to anything, because that doesn't matter. They are located on different floors. You are asked to manually evaluate if switch 1 is on and switch 2 is on. You are on the same floor as switch 1, and can see that it happens to be off. In this particular scenario, do you need to go check the other switch to know if your condition is true or false? Of course not.

If on the other hand, that first switch was on, then you would need to check the second one. But in this scenario the first one is off. Is it ever possible for the condition to become true if nobody flips switch 1 to on? No, obviously not. You only need to check/start trying to keep track of the state of switch 2 once switch one is flipped on.

Basically we are taking advantage of the mathematical fact that in a boolean expression of the form: A(x) ∧ B(y), where A and B are Boolean functions, for any possible value of r, s, t, and u, if A(r) is false, and thus A(r) ∧ B(s) is false, A(t) ∧ B(u) will also be false if r=t regardless of what differences might exist between s and u. This is because if r=t then A(r)=A(t), and we know A(r) is false, so false=A(t).

Similar but different logic applies to the the OR case.

[github-actions[bot]]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.