Drop the auto generated groups.

balloob commented 5 years ago

I want to use this issue to explore if there is still a use case for auto generated groups, and if so, what we can add to fix that.

Auto generated groups are the groups maintained by entity components. These groups are hidden in the UI by default. However, on every state change to an entity of things that they group, they will calculate if their state is still up to date.

This issue is not talking about dropping groups, this issue is about dropping the automatically generated light.all_lights group.

Use case: control all lights at once

This can be done by specifying entity_id: all to light.turn_on.

Use case: I don't have to do anything to maintain a group

Not a good use case. Devices are not added/removed on a daily basis, it's fine to maintain a custom group.

Use case: see if someone is home

This should be replaced by a new feature built on top of persons. Maybe new person.anyone and person.everyone entities.

Use case: card with all automations in frontend

No alternative besides manual curation. Maybe we need an automation card? Or is this something we also need for lights?

SeanPM5 commented 5 years ago

One use case where group.all_lights might still be needed is in scenes. entity_id: all wouldn't work because scenes can have other types of entities besides just lights.

scene:
  - name: All Off
    entities:
      group.all_lights:
        state: off
        transition: 3

balloob commented 5 years ago

What would be the benefit of keeping that as a scene and not replace it with a service call ?

DubhAd commented 5 years ago

Use case: see if someone is home

The auto generated group is already a bad choice for this, because once you're using anything like a general network or bluetooth tracker, the group includes many "random" devices.

For the other use cases, there's already a glob for customize. How about a glob for groups? That would allow those who really want this to continue to do it.

DavidFW1960 commented 5 years ago

Just FWIW, re: automations auto generated group. I just switched to this in lovelace to not use group.all_automations anymore:

          - type: entities
            title: Automations & Scenes
            show_header_toggle: false
            entities: 
              - type: custom:auto-entities
                card:
                  type: custom:fold-entity-row
                  head:
                    type: section
                    label: All Automations
                  open: false
                filter:
                  include:
                    - entity_id: "automation.*"

balloob commented 5 years ago

@DubhAd you actually hit the nail on the head: the groups seem nice in the beginning, but as your system grow, they end up not tracking the things you want and are not customizable.

@DavidFW1960 if we add enabled status and last trigger to the automation config panel, would you still want to see your card?

DavidFW1960 commented 5 years ago

Do you mean in the Configuration->Automations with the GUI editor for automations? If so, yes. I display these on my switches page as I have some automations that are not enabled and I trigger them manually when I need them... I can also enable/disable automations from there as well as see what they were last triggered...

Regarding lights and switches.. I show them on cards and manually add them. I don't need an all switches or all lights group..

pvizeli commented 5 years ago

I never used this automated group. Anyway, a bit more love to our scene component would be not to bad in the future. So you can solve that i.e. as:

scene:
  - name: All Off
    entities:
      light:
        state: off
        transition: 3

but that is offtopic

DavidFW1960 commented 5 years ago

Just another thought as well @balloob .... the GUI Automations editor doesn't show automations contained in packages....

Swamp-Ig commented 5 years ago

Would it be worth having "template groups" to replace it? These could be specified as a regexp / glob / general template to pick up entities by ID. I am happy to contribute such a thing after 1.0 if desired, although it would need home-assistant/home-assistant#22587 through to completion. I can write this up as an arch issue if wanted.

Mariusthvdb commented 5 years ago

Must admit I feel for the basic logic in the first post a lot. However I find good use for at least 3 all_groups:

automations: great to have at hand and see if restore state did its job correctly, Also great for debugging unexpected behavior in the system, of course especially when editing automations. scripts: cool to have at hand, to see if all is going well, especially with scripts using delay or other switchable services. Great debugging tool lights: Since several components create lights that aren't manually added to the config (Hue to name but one) it is very convenient to have this automatic group to be able to check the internal HA light states.

Please allow for creation of these, (others may find good use for the other groups) by some setting in the config. Ive suggested in https://github.com/home-assistant/home-assistant/pull/23789#issuecomment-491173254:

create_all_groups: True

defauting to False of course.

if that won't be within reach, somehow @Swamp-Ig 's suggestion for a template groups would be very very great to have, and that would be of immense use in other use cases. Just today tried:

group:
  media_players_on:
    name: Media players on
    control: hidden
    entities:
      - >
        {{states('sensor.media_players_on')}}

with the template being a list of media_players. This would be a magic addition to HA, and allow for an all_group substitution also!

hope this adds to the discussion, thanks for considering

balloob commented 5 years ago

@Mariusthvdb your first two use cases should get covered by the automations and script panels.

create lights that aren't manually added to the config

I don't understand this one

Mariusthvdb commented 5 years ago

create lights that aren't manually added to the config I don't understand this one

Hi, thanks. Hue creates lights for entertainment groups eg that aren't in the users sight without having a look at the all_lights group. put differently: there are many more entities in the all_lights group than actual lights. next to that, HA Hue integration creates lights entities for Hue Rooms and now Hue Zones.

These are all displayed in the all_lights group while they aren't in fact single lights, and the casual user would not now of their existence without that all_lights group.

Would probably be outside the scope of this topic to suggest maybe having a look at that aspet of Hue integration. Still also the Tradfri integration has

Must admit I am talking regular HA, not Lovelace here, but if the automations and script panels you are referring to could be expanded with a lights panel, that would of course be great too.

Next to that, the suggestion for a user made template group would be perfect, and a solution to all requests

balloob commented 5 years ago

Well isn't that what unused entities is for though?

Mariusthvdb commented 5 years ago

I guess unused entities approaches that, but you have to admit that is a far larger and mixed group, not as concise as the all_lights group would be... and, only for Lovelace.

What do you feel about the template group? If that would become available, magic.

iantrich commented 5 years ago

Lovelace is the future ;)

MartinHjelmare commented 5 years ago

Lovelace is now!

Swamp-Ig commented 5 years ago

Lovelace groups are great, but they're front-end only so they're not usable in automations. I will write up my thoughts on it shortly.

Mariusthvdb commented 5 years ago

Lovelace groups are great, but they're front-end only so they're not usable in automations

No only that, but I fear, (and this is a more general observation, so sorry if this drifts a bit from the original topic), that because of that Lovelace is causing more stress on the system than 'regular' HA. This is particularly noticeable with the Hue integration where reloading Lovelace causes these lights to go unavailable. Long story, well commented in other threads and discussions.

Nevertheless, going back to Post 1 in this thread where Balloob says the reason for dropping auto generated groups is amongst others:

on every state change to an entity of things that they group, they will calculate if their state is still up > to date.

I understand this as the desire to take out as much unnecessary strain from the system as possible. Which all agree on I suppose. But taking these auto groups out might not be priority nr 1, and having an extra inspection why Lovelace is so much more heavy and less responsive than regular HA might be more worth the while.

Swamp-Ig commented 5 years ago

Would it make some kind of sense for groups to not track the state of their members, but rather just be used for grouping things alone?

Really AFAICS the state tracking was just there for pre-lovelace, when the group's state was displayed in the UI.

You could still delegate turn_on and turn_off to the group members, but just not track their state.

Mariusthvdb commented 5 years ago

but just not track their state

at least in my daily usecase this is what I use the all_groups for exactly.... so please don't take that out in an alternative... please don't take out at all, unless substitute with the template group, which would solve all:

allow for the use of tailor made groups (including dedicated all_groups, but have a plethora of other options now unavailable)
minimize strain on the system (no default component auto groups)
all of OP 's objectives.

balloob commented 5 years ago

Lovelace is the UI. The legacy UI is no longer maintained and will eventually be dropped when it is gets too broken/old.

This is the first time that I hear that Lovelace is causing issues in the event loop, probably caused by the configuration being very big, causing JSON serialization to be intense (we had a similar issue with logbook+history in the past). Please open an issue and tag me.

However, it is not going to stop us from moving forward with this plan. This is tagged for 1.0. This is what we will add before removing them:

add an all_person group to track the person integration.
add enable/disable toggles to automation config panel, show non-editable automations, allow triggering them or open more info
show non-editable scripts on script config panel, allow triggering them or open more info

balloob commented 5 years ago

If you want to see a list of all lights in the frontend, you can either maintain one yourself, or use the monster card with a light domain filter. I consider this a UI concern, and not something that should impact backend core design.

Mariusthvdb commented 5 years ago

will return to your point of an issue for Lovelace s responsiveness, and yes, I had the same issues with history/logbook. Fear to use these is still there, because it would hang the instance, forcing a hard reset (pulling the plug)

about the Monstercard: I've issued a request there already december 2018 for using the templates. No response yet.

Would like to maintain that in this discussion. Group templates: If added, all options would be open to the user, and none of the issues you address would arise any longer.

iantrich commented 5 years ago

Monster card is no longer actively maintained. Might want to check auto-entities from @thomasloven. On that same note, I am in the process of bringing over monster card and others. I've been trying to work with @ciotlosm but he's pretty busy these days and I'll probably just fork them this weekend to custom-card org

Mariusthvdb commented 5 years ago

Thanks for that suggestion and chiming in! Will certainly do that.

Still, as other posters have already marked, Lovelace is all frontend, while having a solid group template would be preferable.

It wouldn’t force us to do all sorts of customizing in the frontend, resulting in a much cleaner configuration imho.

Swamp-Ig commented 5 years ago

Ok have written up template groups. I probably won't have time to do this prior to 1.0, but it's a good replacement.

Mariusthvdb commented 5 years ago

@Swamp-Ig wonderful, hope you can make it! If you need some testing, please let us know so we could assist

jgalak commented 5 years ago

Just wanted to say that I came here to advocate for some sort of replacement for the all_automations use case, but the suggestion in https://github.com/home-assistant/architecture/issues/177#issuecomment-474637844 completely addresses my use case, thank you.

CCOSTAN commented 5 years ago

I guess conditions and triggers that leverage group.all_* would need to be converted to custom manual groups. https://github.com/CCOSTAN/Home-AssistantConfig/blob/master/config/automation/guard_dog.yaml#L16-L19

https://github.com/CCOSTAN/Home-AssistantConfig/blob/master/config/automation/late_night_outside_helper.yaml#L7-L10

as well as some of the speech jinja stuff. https://github.com/CCOSTAN/Home-AssistantConfig/blob/master/config/script/speech_engine.yaml#L78-L101

I guess it's not too big of a deal. BTW: had no idea about the all parameter you could use with entity_id for services. that's super awesome! :)

brianhanifin commented 5 years ago

For those that are looking for a way to create a replacement for group.all_lights, the following script dynamically generates a group I named group.every_light. Clicking on group.every_light in the dev-state list appears to reproduce group.all_lights perfectly.

script:
  create_every_light_group:
    sequence:
      - service: group.set
        data_template:
          object_id: "every_light"
          entities: >
            {{ states.light | map(attribute='entity_id') | join(',') }}

lindsaymarkward commented 4 years ago

@balloob - I'm not sure if I should post here or on issue https://github.com/home-assistant/home-assistant/issues/30912
Can we get the all groups back? I have read through issue and don't see the motivation. Is the problem that:

on every state change to an entity of things that they group, they will calculate if their state is still up to date.

Does that cause a big performance penalty? People would be happier with their loss if they knew what they were gaining in return. Like https://github.com/home-assistant/home-assistant/issues/30912#issuecomment-576892203 I have multiple cases where I use an all group including checking if any light is on when I leave home. I also have a simple "sleep" scene that just says all lights should be off. This used to be one line: group.all_lights: off Now it can't be a scene (scenes are easier to think of than scripts), so it has to be a script with a service call, which is harder to figure out and more code to write. (answering your question in https://github.com/home-assistant/architecture/issues/177#issuecomment-474200871)

Could you or someone else who knows tell us what the benefit of this breaking change is here? Thank you.

balloob commented 4 years ago

They are not coming back. If you want the logic back, feel free to create a custom component. Reasons for the change are easily found, I am not going to repeat myself.

iantrich commented 4 years ago

There are already a few appdaemon scripts for it

brianhanifin commented 4 years ago

@lindsaymarkward I just created a topic on the forum: How to survive without the "ALL groups". Myself and many others would be happy to help you make this transition. Together I know we can come up with a solution that will work for you.

lindsaymarkward commented 4 years ago

Thanks @brianhanifin for this reply and for responding to my specific question about scene in your forum post... But that's not my point/question. I'm asking for a community that's a bit more helpful in terms of understanding "why". I've already got a solution thanks to the "PSA" post in the forums (thanks @petro), but I've yet to see a clear explanation of why this move is a step forward. Paulus said the reasons are easily found, but I'm not finding it easy. The release notes say only:

Furthermore, time to say bye-bye, to all generated entity groups (group.all_*). Chances are, you have never heard of those because they were hidden.

Nothing there about why. This issue says:

on every state change to an entity of things that they group, they will calculate if their state is still up to date.

Is that the reason? I asked, but did not get a reply to this question. It would be great to be able to understand, and as I said, it would help those who have to make changes to do so with a sense of why it's better now.

Another challenge is how easy it is to find help with this sort of thing. It seems more reactive than proactive and you have to search in multiple places (release notes, issue, forums). I'd love to see a link to the help in the release notes (they can be updated after publishing). Thank you.

brianhanifin commented 4 years ago

@lindsaymarkward While I am not a Home Assistant developer, I have been programming for work or hobby for many years. So I believe I can provide a reasoned answer as to why.

For every piece of software there is a constant struggle between features and processing time. I don't know how long you've been using Home Assistant, but before Lovelace came along pages were generated by groups of entities. The whole UI was centered around groups and subgroups of groups.

Groups also did, and still do aggregate the states of all of the entities in the group. For example, a group of device_trackers can let us know if "anyone" is home. Every time one of the device_trackers in that group changes state, a process triggers to calculate if at least one device_tracker is "home" or if all of them are "not_home". When you get a house full of lights, switches, sensors, binary senors, etc... that is a lot of extra processing time being spent on calculations that may not make sense to be enabled by default. Especially when we want the software to run well on low spec computing devices like the Raspberry Pi.

In this case, the functionality is available in other ways, so we didn't lose functionality so much as we have to learn a new way to accomplish the same thing. For those that want to recreate those groups, there are several ways to do it. Personally I have a script which creates a group of lights which are turned on, so an automation I have setup won't accidentally turn on the bedroom light at night. I run the script to refresh that group right before I need it. That way my group is only being refreshed on demand, instead of constantly. And, I don't have half a dozen or so other groups being refreshed constantly in the background even though I don't have any use for them.

I hope this helps you understand some of the reasoning that can be behind a tough decision such as this.

P.S. I was prepared for this particular change as I have seen various discussions on this topic for months. I have been frustrated by an unexpected breaking change on many occasions before, so I understand your disappointment.

mueslo commented 3 years ago

There still doesn't seem to be a viable replacement for "group.all_persons" / "persons.anyone".

I want to trigger an automation when everyone leaves the house/anyone returns home. The only way I can see this being accomplished currently is either 1) by adding a template trigger with "{{ states.person | selectattr("state", 'equalto', 'home') | list | count == 0 }}". If I'm not mistaken, this means the template must be constantly evaluated. Something more event driven would be nicer. 2) The alternative would be adding a custom "everyone" group. But since groups cannot be managed in the front-end, this is tedious, since my instance frequently changes or adds new people, and it would be preferable to include them automatically.

home-assistant / architecture