add optional time tracking possibilities in templates

[Mariusthvdb]

Context

The issue motivating this decision, and any context that influences or constrains the decision. Your motivation cannot include that the frontend is not representing the data correctly, as that is a frontend issue.

coming from https://github.com/home-assistant/core/pull/41147 where time tracking in templates is discussed, which now is closed, at updating per minute for sensors using now(). replacing the formerly used {% set x = states('sensor.time') %}

Please consider the following, where updating per minute simply is not necessary and remains costly. We now can no longer update these on demand, either per automation, or per other triggering entity. Which was a very fine option, next to the ultimate intelligence of the new template engine

Proposal

add the option for eg {% set x = states('sensor.date') %} for sensors that only need daily updating, and, preferably, an option to have the templates not update at all, so we can update upon triggering an automation/script

The change that you're proposing.

Consequences

What becomes easier or more difficult to do and any risks introduced by the change that will need to be mitigated.

This would at least for the daily sensors prevent an extra 1439 updates a day per sensor. In decreasing the processor usage as much as possible, which is very much needed on the bigger instances, each update less counts.

Second to that, this would give ultimate (and optional!) control to the user, who is in need for that. Not used per default, which is probably fine now with the latest PR. But, as an extra setting.

This might even be a great addition to the other templates as well, having the option to not update at all (allowing on demand) or per single other entity, like {% set x = states('sensor.date') %}

[frenck]

So, why should we try doing that hacks in templates, while we have automations?

For those special cases, automation can be used to trigger and update to ones custom liking, right?

PS: Please note, an architectural discussion issue opened, means you are implementing it? This is not a feature request forum.

[Mariusthvdb]

not suggesting we 'hack' in templates. Up to 116 this is still current and an advised (by several devs in the community) tool. Dont really understand why you would call that hacking.

And no, not right: we can not use the automation , because the templates updates continuously now. Well, strictly speaking we can of course update any entity, but that is now futile, since the merged PR will update templates each minute. Which is definitely overkill (for some)

As for the implementing: again why yes of course, as we have been implementing these for HA iterations up to now..

This is not a feature request forum.

I was asked to deliberately open this, as linked in the opening sentence. so please don't be so aggressive

[frenck]

I called it a "hack", because it introduces control options of entity behavior inside templates (please note, behavioral changes, not to be confused with the values a template results in).

This is new, as general all behavior of an entity is defined in the YAML configuration. Doing that, might be fine, but has a wider implication. For example, a template is not just used to define entity values.

IMHO this is not wished for, as it would create a second dimension (third if one considers the UI to be one as well), that defines entity behavior.

I was asked to deliberately open this, as linked in the opening sentence. so please don't be so aggressive

I didn't mean to be aggressive, it is just you are not the obvious person to implement this architectural change. Hence me stating that. If that offended you, I'm sorry.

[tdejneka]

it introduces control options of entity behavior inside templates

Actually it is asking to restore control options that had always existed but were eliminated in 0.115. So the architectural discussion is not to introduce something new but to re-establish something we always had (in Template entities).

What was once available was the ability to control the evaluation frequency of any Template Sensor. The two most-used frequencies were once a minute and once a day.

Regardless of the entities present in the template, the template was evaluated at a specific frequency (once a minute or once a day) and not when one the template's entities changed state. This was a very resource-efficient way of creating Template Sensors, possibly with complex templates, that only needed to be evaluated infrequently (for example, once a day, at the start of the day, is typically used to reset a daily calculation).

Bdraco's recent PR to allow now() to evaluate the template once a minute is a welcome improvement, However, if I understand it correctly, it doesn't exclude the template from being evaluated by the state-changes of other entities within the template. In other words, it doesn't restore the degree of control we used to have nor does it restore the lost ability to evaluate the template once a day.

What was once very easy to do has been lost. The suggested workaround, to use an automation with a Time Trigger (to force evaluation of a Template Sensor) overlooks the fact that the sensor's template continues to be evaluated according to the new rules and not exclusively by the Time Trigger. There's simply no easy way now to ensure a Template Sensor is evaluated only once per day.

What Mariusthvdb is proposing is that this lost ability be restored or at least some means of emulating it, short of having the user create a python_script to achieve it.

[frenck]

What Mariusthvdb is proposing is that this lost ability be restored or at least some means of emulating it,

Emulating can be done using automations.

While I get it didn't become easier for this specific use case, the general use of a template for the casual / most used use-cases did become easier. I think we have to accept we cannot always cover all use-cases to stay the same at this stage of the project.

Edit: I might be backfiring too much, however, a concrete example of the end result isn't provided either at this point.

[balloob]

Please consider the following, where updating per minute simply is not necessary and remains costly.

(emphasis mine)

Do you have any data or logs to show how the cost of these templates are negatively impacting your system?

Without data we'll just end up discussing hypotheses, which makes no sense.

[Mariusthvdb]

Well, tbh, I am in this cost reducing mode ever since 115 introduced the new template engine and the system came to a stand still because of the continuous updating of only 2 templates.

Upon request by Bdraco, we tried and are still trying to rewrite as many templates as could be using the new techniques, and to prevent unnecessary updating templates.

Also, to offer examples of templating challenges that would profit of reducing/controlling the updating.

I'd say, this would at least fall into that category, updating 1440 times a day while only 1 would suffice.

Add a few of those, and they are available, to get the picture.

Why spend power if not necessary. Even if available?

And, as said by Taras above, this is not asking for introducing something new. On the contrary, I might add.

On the emulating mentioned by Frenck, I honestly don't understand what that means. We simply can not control the updating frequency anymore.

Which is a real letdown for any user oriented automation system imho.

Even though I managed to rewrite several Jinja templates to Python scripts which I now update by an automation, on demand, or once a day.

And which simply worked before 115 by adding 1 line to a Jinja template...

[balloob]

When you have collected the data to show it negatively impacts performance, please create a bug report (not an architecture issue).

Going to close this topic now.

[amelchio]

Emulating can be done using automations.

It really can't. The update frequency can only be increased with automations while this issue is proposing to allow a decrease in the frequency (all the way down to no automatic updates at all).

That said, I agree that the proposal is lacking concrete examples that are hurt by the way it currently works (i.e. in 0.116).

[frenck]

It really can't. The update frequency can only be increased with automations while this issue is proposing to allow a decrease in the frequency (all the way down to no automatic updates at all).

That is incorrect. An automation can calculate/have the template and write it into an input_* helper. As an automation can define time-based (and time-based patterns) as triggers, that would be a viable workaround for such a case.

Depending on how one triggers an automation, it becomes an increase or decrease (but that is the control one has in automations).

[Mariusthvdb]

sorry Frenck,

but if you would have at least had given it a try, you would have been able to read many threads ago that the automation/input_helper is in no way a solution, because the input helper has a limit of 255 characters.

This has been gone over time and time again, and considered not a viable solution by the dev team currently on this subject.

Besides that, this issue was not on a combo like that, but on templates. Your solution is a workaround for the demise of control we had up to 115. (with the new PR merged for 117) this will be more evident, since the small bit of control we had left will be gone too.

Which was the start of the whole excercise to begin with.

this thread is full of examples: https://community.home-assistant.io/t/heads-up-upcoming-breaking-change-in-the-template-integration

[amelchio]

That is incorrect. An automation can calculate/have the template and write it into an input_* helper.

It really isn't. An input helper cannot do everything a template sensor can do, such as be iterated by states.sensor. It also has a different UI which might not be desirable.

You cannot even make a hidden input helper and then have a template sensor update from that because the input helper is constrained by a state size limit while sensor attributes are not.

So there is no workaround but that does not mean that there is an issue in the first place ... data is still lacking.

[frenck]

The size limit was not scoped in this architecture issue so not part of my consideration.

[tdejneka]

I have about two dozen examples of Template Sensors that use sensor.date. However, I'll just limit it to two simple examples (given that this topic is now officially closed for discussion).

Prior to 0.115 this was updated once a day and that was sufficient:

      days_until_monday:
        entity_id: sensor.date
        value_template: >
          {{ (7 - now().weekday()) % 7 }}

In 0.116, a small modification was needed to keep it updating just once a day:

      days_until_monday:
        value_template: >
          {% set x = states('sensor.date') %}
          {{ (7 - now().weekday()) % 7 }}

In 0.117, it appears there will be no way to constrain it from being updated 24x60=1440 times a day:

      days_until_monday:
        value_template: >
          {{ (7 - now().weekday()) % 7 }}

Here's one that calculates remaining days until a birthday. Once again, this will be evaluated 1440 times instead of just once (representing 1439 needless daily evaluations of the template).

      event_birthday_john:
        friendly_name: "John's birthday"
        value_template: >
          {% set mn = 8 %}
          {% set dy = 1 %}
          {% set yr = now().year if now().month <= mn else now().year + 1 %}
          {% set d = now().replace(year=yr, month=mn, day=dy, hour=0, minute=0, second=0, microsecond=0).timestamp() %}
          {% set n = now().date() | as_timestamp %}
          {{ ((d - n) // 86400) | int }}

The most discussed example was the one concerning a Template Sensor that reported all unavailable sensors. It was a prime example of what was achievable prior to 0.115 but not afterwards (it had to be re-written as a python_script).

[Mariusthvdb]

yes, exactly. And those are not merely futile examples, but simply truly used in the config. as are these:

      dayofyear:
        friendly_name: Day number
        value_template: >
          {% set trigger = states('sensor.date') %}
          {{now().strftime('%-j')}}

      weekofyear:
        friendly_name: Week number
        value_template: >
          {% set trigger = states('sensor.date') %}
          {{now().strftime('%-U')}}

      dag:
        friendly_name: Dag
        value_template: >
          {% set trigger = states('sensor.date') %}
          {% set dagen =
            { 'Mon': 'Maandag',
              'Tue': 'Dinsdag',
              'Wed': 'Woensdag',
              'Thu': 'Donderdag',
              'Fri': 'Vrijdag',
              'Sat': 'Zaterdag',
              'Sun': 'Zondag'} %}
          {% set state = now().strftime('%a') %}
          {% set dag = dagen[state] if state in dagen else state %}
          {{dag}}

      maand:
        friendly_name: Maand
        value_template: >
          {% set trigger = states('sensor.date') %}
          {% set maanden =
            { '01': 'Januari',
              '02': 'Februari',
              '03': 'Maart',
              '04': 'April',
              '05': 'Mei',
              '06': 'Juni',
              '07': 'Juli',
              '08': 'Augustus',
              '09': 'September',
              '10': 'Oktober',
              '11': 'November',
              '12': 'December',} %}
          {% set state = now().strftime('%m') %}
          {% set maand = maanden[state] if state in maanden else state %}
          {{maand}}

      vandaag:
        friendly_name: Vandaag
        value_template: >
          {% set trigger = states('sensor.date') %}
          {{states('sensor.dag')}} {{now().strftime('%-d')}} {{states('sensor.maand')}} {{now().strftime('%Y')}}

      today:
        friendly_name: Today
        value_template: >
          {% set trigger = states('sensor.date') %}
          {{now().strftime('%A %-d %B %Y')}}

      days_current_month:
        friendly_name: Days current month
        value_template: >
          {% set trigger = states('sensor.date') %}
          {% set month = now().strftime('%-m') %}
          {% if month in ['1','3','5','7','8','10','12'] %} 31
          {% elif month in ['4','6','9','11'] %} 30
          {% else %} {{'29' if (now().strftime('%-y'))|int//4 == 0 else '28'}}
          {% endif %}

      remaining_days:
        friendly_name: Remaining days
        value_template: >
          {% set trigger = states('sensor.date') %}
          {% set this = now().replace(hour=0).replace(minute=0).replace(second=0).replace(microsecond=0) %}
          {% set next = this.month + 1 if this.month + 1 <= 12 else 1 %}
          {% set last = this.replace(month=next, day=1) %}
          {{(last.date() - this.date()).days}}

      past_days:
        friendly_name: Past days
        value_template: >
          {% set trigger = states('sensor.date') %}
          {% set this = now().replace(hour=0).replace(minute=0).replace(second=0).replace(microsecond=0) %}
          {% set next = this.month + 1 if this.month + 1 <= 12 else 1 %}
          {% set first = this.replace(day=1) %}
          {{(this.date() - first.date()).days}}

I truly hope the final thought/decision for a clean and efficient as possible way of doing this is not writing python scripts for these daily max update sensors.

this alone is worth 12951 unnecessary updates in the system. Per day.

[balloob]

Just to give an idea of what we're talking about here. Here is a simple timing of executing the Python in the first template. It's not accurate because it ignores some of the template handling, but it's a good indicator to what magnitude we're looking at.

❯ python3 -m timeit -s "from datetime import datetime" "(7 - datetime.now().weekday()) % 7"
1000000 loops, best of 5: 333 nsec per loop

It takes Python 333 nanoseconds to calculate the days until Monday.

333 * 1339 = 479 187 nanoseconds.

There are 1 000 000 000 nanoseconds in a second so this is equivalent to 0.0005 seconds.

For this example, let's assume that the template handling in Home Assistant is horribly inefficient and it makes it 100x as slow. Now this template will take up 0.05 seconds per day.

Everyone here has already spend more time on discussing this topic than that these templates are going to be running in our lifetimes.

I am going to unsubscribe from this topic now. Feel free to tag me if you have data.

[amelchio]

Seems like balloob is not using a Pi :-)

You can measure your templates in the Developer Tools by wrapping in now():

{% set before = now() %}
          {% set trigger = states('sensor.date') %}
          {% set this = now().replace(hour=0).replace(minute=0).replace(second=0).replace(microsecond=0) %}
          {% set next = this.month + 1 if this.month + 1 <= 12 else 1 %}
          {% set last = this.replace(month=next, day=1) %}
          {{(last.date() - this.date()).days}}
Took: {{ now()-before }}

this alone is worth 12951 unnecessary updates in the system. Per day.

I am also not using a Pi but the above took 0:00:00.000164 for me so I can render about 6000 of those. Per second. Doing 9 redundant ones per minute cannot be measured, it will be lost in the noise.

[Mariusthvdb]

though fully appreciative of the above calculations, isn't this about reducing as much as possible? Per se. considering these few template are not the only thing going on in the system (leading up to a mean 11 % usage on my Rpi4) shouldn't we reduce as much as possible wherever we can.

Again, I am appreciative of spending most effort where it most counts (the famous 80/20, or even 90/10), and, with your and Bdraco;s help try to do so in all possible fields, controlling updating of templates (like we were able to) should be an easy win. Might be small, but there's a couple of sayings about that. I would think in any language ;-)

about the template above: on my Pi4 it just

Took: 0:00:00.001210

so, again, your mileage may vary. still small, but a factor 10 larger....

[tdejneka]

isn't this about reducing as much as possible?

Apparently not. The development team lead isn't concerned with it so we are wasting our time expressing concern for lost control and efficiency.

This philosophy of "it's only nanoseconds" is what brought your RPi3 to a standstill in 0.115 (while the same template in 0.114 worked effortlessly). However, the people responsible for it don't think that's a big deal (classified as 'edge cases') so neither should we. You'll just have to stop creating 'edge case' templates ...

[amelchio]

The size limit was not scoped in this architecture issue so not part of my consideration.

So do you now agree that there is no general workaround for this proposal?

I mean, it's fine to turn proposals down but it should be done for valid reasons. I think "there is no performance issue" is valid here but "there is a workaround" is unfortunately not.

This philosophy of "it's only nanoseconds" is what brought your RPi3 to a standstill in 0.115 (while the same template in 0.114 worked effortlessly).

The standstills were not ignored, they were addressed with the rate limits introduced in 0.116. I think the scare from those few templates has caused a general fear of updating templates. But updates are usually not very costly compared to all the other things Home Assistant is doing.

It is also arguable that it worked effortlessly before. One had to make the template, wonder why it didn't work, read the manual, configure the time sensor and then hack in the redundant sensor.time reference.

However, the people responsible for it don't think that's a big deal (classified as 'edge cases') so neither should we. You'll just have to stop creating 'edge case' templates ...

No, you do not have to stop creating them. You have to start showing them. Preferably with performance measurements but both bdraco and I have so far been willing to test things on our own when we actually get something to test.

If no performance problem can be demonstrated, right ... you will have a hard time convincing anyone to add complexity.

[Mariusthvdb]

I find this last comment a very constructive and helping one, so thanks for that.

The thing is though, it is very very difficult to 'measure' the cost of a single template in a Homeassisant OS system, which doesnt have a monitoring system like py-spy available.

I have been trying to install that in Portainer but even that isn't very straightforward, and it was claimed that wasn't measuring the processor usage in the first place, so I refrained from further trying.

So, it seems you have built in a wall that can't be taken.

"Proof us you're having problems, or we won't listen." But the system itself doesn't allow for the methods to proof.

Of course, this is embracing the new way of going forward with HA. As was phrased elsewhere:

ignore the wishes of long term users for powerful control of their system in preference to dumbing down the available options to suit a wider audience.

let me add:

wishes for powerful control of their system that was available up to 115.

so you may understand these users are not embracing this new way. And let me counter: the dev team will have a hard time convincing anyone, taking out control in favor of (yet another quote of a well respected community member):

is that it is simple to use so a lot less people will have to understand it now.

is the way to develop, for the most advanced Home automation system around. Do away with (fine grained) end user control, and instead use semi-brute-force.

This architectural discussion apparently has to stop. I respect that.

I most fervently hope this will not be the final decision though.

As an end user I expect to be able to take control, and not be left to the Rise of the Machines.

[dgomes]

Jumping into a heated discussion...

I think that sensor.date is the root cause of this discussion (I might be miles away from this issue, coming with this conclusion)

Can I ask if a binary sensor, lets called it "binary.alarm", which could be configured with whatever periodicity the user likes would make life easier for everyone running these "performance sensitive" templates ?

[thomasloven]

I'm sorry, but parts of this discussion seem outright ridiculous.

Much of the problem seems to be that you can't get the date once per day because your method to get the date relies on now(), and your workaround of choice is to use sensor.date to make the recalculation of the date only happen once per day? Sensor.date?

{% set trigger = states('sensor.date') %}
{% set this = now().replace(hour=0).replace(minute=0).replace(second=0).replace(microsecond=0) %}

{% set this = strptime(states('sensor.date'), "%Y-%m-%d") %}

I understand that's "just an example", but all this example illustrates to me is how no one is prepared to take a step back and look at how the new changes could possibly improve the situation.

[Mariusthvdb]

Hey Thomas,

Not exactly sure who you are addressing here, but I feel spoken to ;-) And I would pride myself to always be openminded, and for sure, willing to find out 'how the new changes could possibly improve the situation.'

having said that, how would you alter this specific template going forward so it updates once a day?

remember, this one is from the period in HA where we needed to use a trigger to update any template using now(), which didn't update at all. Using sensor.date was sufficient so no need to speedup/increase the updating frequency.

the alternative was brought by Amelchio.

Coming 117 there is no longer any need for an external trigger since templates using now() will update once a minute. Which, for this exercise, would be too frequent.

[tdejneka]

No, you do not have to stop creating them. You have to start showing them.

The 'edge case' I was referring to was the 'unavailability sensor'. In a recent forum post you conceded it could not be 'fixed'. Currently, the solution is to re-design it as a combination of a dynamically generated group and an automation or as a python_script. What used to be achievable with a Template Sensor now requires a more complicated solution.

I realize that some design changes involve 'casualties' (the 'unavailability sensor' being a prime example) but it's been a struggle for others to concede to the same fact. Overall, the new technique is welcome but it has broken some long-established design patterns the community has employed with no easy substitute. The gist of this thread is to find (or create) those substitutes.

Much of the problem seems to be that you can't get the date once per day because your method to get the date relies on now(), and your workaround of choice is to use sensor.date to make the recalculation of the date only happen once per day?

Prior to 0.117, the use of now() would not cause the template to be updated periodically. As a workaround, sensor.time or sensor.date was tossed into entity_id, or directly into the template, to ensure the template was evaluated either once a minute or once a day.

The choice of which one to use, sensor.time or sensor.date, was largely based on the template's purpose. If it performs a date-related calculation, in days, weeks, or months, then sensor.date would be the more efficient choice because it limits the template's evaluation to once a day. Otherwise, you'd use sensor.time to ensure per-minute evaluations.

In 0.117, the now() function becomes a first-class citizen and ensures per-minute evaluations of the template. It means you no longer need to include sensor.time, or sensor.date, because the template is guaranteed to update every minute. For most users, that's a tremendous improvement; I've lost count of how many times (over the past 2 years) I've explained to users that the reason their Template Sensor fails to update is because it contains now() and no other entities.

However, the flip side is that one will no longer be able to constrain a template's evaluation to once a day. Just about any date or time-related calculation uses now() (often to get the current timestamp). However, in 0.117, its mere presence within the template ensures the template gets evaluated every minute. Therefore the template will be evaluated at least 24x60=1440 times a day instead of just once. However, the development team feels this is not a significant daily processing increase and so the issue is considered to be moot.

[thomasloven]

Win-win-win

The unavailability sensor sounds like a perfect candidate for an automation that runs once per day and sets an input_number.

[tomlut]

So to prevent my many template sensors like this from updating 1439 more times than needed a day:

- platform: template
  sensors:
    zone_1_day_active:
      friendly_name: Irrigation Day Active
      value_template: >-
        {% set update = states('sensor.date') %}
        {{ ( is_state('input_boolean.zone_1_mon', 'on') and now().weekday() == 0 )
        or ( is_state('input_boolean.zone_1_tue', 'on') and now().weekday() == 1 )
        or ( is_state('input_boolean.zone_1_wed', 'on') and now().weekday() == 2 )
        or ( is_state('input_boolean.zone_1_thu', 'on') and now().weekday() == 3 )
        or ( is_state('input_boolean.zone_1_fri', 'on') and now().weekday() == 4 )
        or ( is_state('input_boolean.zone_1_sat', 'on') and now().weekday() == 5 )
        or ( is_state('input_boolean.zone_1_sun', 'on') and now().weekday() == 6 ) }}

I'm just going to replace them with this:

- platform: template
  sensors:
    zone_1_day_active:
      friendly_name: Irrigation Day Active
      value_template: >-
        {% set weekday = as_timestamp(states('sensor.date'))|timestamp_custom('%w') %}
        {{ ( is_state('input_boolean.zone_1_mon', 'on') and weekday == 1 )
        or ( is_state('input_boolean.zone_1_tue', 'on') and weekday == 2 )
        or ( is_state('input_boolean.zone_1_wed', 'on') and weekday == 3 )
        or ( is_state('input_boolean.zone_1_thu', 'on') and weekday == 4 )
        or ( is_state('input_boolean.zone_1_fri', 'on') and weekday == 5 )
        or ( is_state('input_boolean.zone_1_sat', 'on') and weekday == 6 )
        or ( is_state('input_boolean.zone_1_sun', 'on') and weekday == 0 ) }}

[amelchio]

The 'edge case' I was referring to was the 'unavailability sensor'. In a recent forum post you conceded it could not be 'fixed'. Currently, the solution is to re-design it as a combination of a dynamically generated group and an automation or as a python_script. What used to be achievable with a Template Sensor now requires a more complicated solution.

Eh, no. That was in a response to somebody wanting it to work like in 0.115 – immediate updates, no rate limit. This can be done, but not by tweaking the sensor itself. That is exactly like it used to be in 0.114.

If you are happy with updates once per minute (like everybody apparently was before 0.115), the unavailability sensor works fine by default and no longer needs hand holding (i.e. an explicit sensor.time).

(For reference, my post is here).