Allow for nesting of structs / Put struct definitions in own folder instead of etc/struct.yaml

[onkelandy]

I would prefer to put separate struct files in a struct folder instead of putting everything into one file. Would make things better organised if you heavily use the struct feature. Furthermore it would be consistent to the items, logics and scenes folders.

Going through the files separately at startup might even make it possible to use structs in structs what would be pretty cool in my opinion. For example:

file: struct/1_struct.yaml:

light_actions:
    on_enter:
        se_action_bwm_trigger:
          - 'function: set'
          - "to: eval:1 if stateengine_eval.get_relative_itemvalue('...sa') == 1 and (sh.return_item(stateengine_eval.get_relative_itemid('..stateengine_name')).age() <=4 or sh.return_item(stateengine_eval.get_relative_itemid('..settings_edited')).age() <=4) else 0"
          - 'order: 7'
          - 'delay: 2'

file: struct/2_struct.yaml:

light_rules_lightcurve:
    se_item_lightcurve_active: ..settings.lightcurve.active
    lightcurve:
        struct: light_actions
        enter:
            se_value_lightcurve_active: 'True'

That would make it possible to easily include templates in other templates and therefore make the configuration e.g. for the state engine plugin more flexible and convenient. Of course it would be even better if the order of the definitions and filenames didn't matter.

[msinn]

1. structs are used mainly in plugins (for the time being). I don't think many people are defining many of their own structs. I don't like the idea of having all structs in one file for plugins and a special folder with a bunch of struct definitions in separate files for user defined structs.

   For the average user, the admin interface is the way to go. At the moment, all loaded structs can be viewed seperatly. With the coming versions of the admin interface, an item editor will be added. With the user interface in place, structs can then be a added/managed without the need of fiddling with yaml syntax. With the editor in place, it will become far less important how and where the struct definitions are stored.

2. The possible feature of placing structs in structs will have nothing to do with the matter of storing structs in one ore multiple files. It solely depends on the way the structs are read at startup.

   (btw.: reading structs from multiple files will increase the time it takes SmartHomeNG to start up)

   Startup takes place like this:

   • read the content of the /etc/structs.yaml file in a data structure in memory
   • read the content of the struct sections of every configured plugin into the same data structure
   • read in all item definitions. If an item definitions contains a struct reference, lookup the struct in the loaded struct definitions and replace the reference with the real struct

   Allowing nested structs would require an additional step, that could be timely: After reading all struct definitions, SmartHomeNG would have to iterate through those definitions, replacing struct references with the real structs, until all references are replaced. That would have to take place before the items are loaded, because replacing the struct references in the structs while loading the items would potentially take more time (since structs would/could be used several times in the item definitions).

[onkelandy]

Fully understood. 1.: Okay it's not that important indeed. I just thought it was conclusive to put it in a separate folder as in the etc folder are usually just references to other files. More like a "dictionary" and the actual files are in separate folders. Right now as the feature is new you might be right about the main usage of the structs in plugins. But I am pretty sure the structs have huge potential for standard setups. I'm using them thoroughly in my "redone" items and they are a blast!!!! Will have to write a blog post about it I guess ;)

2.: I thought about replacing structs while reading the structs on the fly but it's not a good solution as order of the definitions would be extremely important. In general it would still be an awesome extension of the template feature.. Is time really an issue? Aren't we talking about some seconds only? It's still in the hands of the user to use it or not. It's more or less the same with the relative items. They also take longer to load in my opinion but they make the setup much easier.

[msinn]

I think the nesting of structs is not too hard to implement, since the iteration can take place as a seperate step after the loading of structs finished and before the loading of items start.

I‘ll put it on the list of features to be possibly implemented in the next release.

[onkelandy]

While you're at it. Maybe you could also look into the possibility to structure templates in hierarchies like you do with standard items? That might make configurations even simpler as you can then decide whether you take the whole structure or just part of it. It also makes editing the yaml file in an editor easier (collapsing, etc.)

[msinn]

@onkelandy Thats not going to happen in the near future. That would be a very time consuming undertaking and there are a lot of features besides structs that need attention.

[thelsing]

A quick hack in case somebody else wants nested structs now:

diff --git a/lib/config.py b/lib/config.py
old mode 100755
new mode 100644
index 4898435d..08599e19
--- a/lib/config.py
+++ b/lib/config.py
@@ -321,6 +321,8 @@ def search_for_struct_in_items(items, template, struct_dict, config, nestlevel=0
                     add_struct_to_template(parent, struct_name, template, struct_dict, instance)
                 if template != {}:
+                    items_template = collections.OrderedDict()
+                    search_for_struct_in_items(template, items_template, struct_dict, config)
                     config = merge(template, config)
                     template = collections.OrderedDict()
             result = True
         else:

[thelsing]

In my setup item parsing is not longer with my change than without it. (On a rpi). So I guess the performance is good enough. Should I add an check for cycles and create a pull request or do you want it implemented after parsing the structs? (My python-foo is probably not good enough to do this.)

[onkelandy]

As bmx stated on Gitter we should wait for msinn for an implementation that respects all new and planned features I guess

[CannonRS]

I think too, that a seperate folder for the structs is useful. :-)

[julianze]

Splitting the structs.yaml in a separate folder would be great. I will use the struct-feature extremely outside the plugins.

My next step is to extract identically item properties (so far copy and paste) from the normal items to one struct for each use case - like jalousie, light switch, heatings .

Then its quite easy to reference and change one property in the struct - which will change all items with this reference .

Nice feature and this gimmick would be quite useful.

[msinn]

Nesting of structs is implemented in Release 1.7.

For the wish of multiple struct files in a seperate folder: What would be the benefit instead having the structs in one file?

If it is for manageability of hundreds of structs, probably a visual editor in shngadmin could fix that. I am no fan of multiple configuration files (except for hundreds to thousands of files like for items).

[julianze]

Thanks for your work! It's quite for better manageability dozens of structs. Item definitions with structs will be more outsorced to structs and only individual parameter will be left in items. At least this would my way of using structs.

A visual editor in shngadmin could solve this. At the moment i am mostly working in the files directly (version 1.5). Migration to latest version will be my next steps now.

[CannonRS]

It's my opinion too. I would like to use more structs and fill only the parameters within the item configuration. If we use separate foldes, we don't have large struct files and looks more clear instead having one large struct file for all definitions.

The question is, why we don't have only one file for the items? ;-)

[msinn]

Because ist has been that way since the original smarthome.py from mknx.

To mehre Structs and substructs from Plungin and one file is complex enough as it is.

A combined struct file will stay smaller as most item files written by users. So it should be managable.

In one of the coming rreleases I will write a gui editor for items and for structs, but I am not going to implement somethong like a structs folder with multiple struct files in it.

[julianze]

In one of the coming rreleases I will write a gui editor for items and for structs, but I am not going to implement somethong like a structs folder with multiple struct files in it.

That´s okay for me. the struct features saves a lot of time, so i can live with one big (in my case) struct file ;)

Thank you for your coding-time ;)

[onkelandy]

How good are the chances to have sub-structs available within a struct hierarchy? Something like

mainstruct1:
    childitem1:
        struct: substruct
mainstruct2:
    childitem2:
        struct: substruct

Would be a huge enhancement for the substruct feature, also in plugins.

A concrete example. I got some lights with just one "color" and some with warm and cold colors. The substruct could look like this:

dimmen:
        knx_dpt: 5001
        visu_acl: rw
        type: num
        database: yes
        influxdb: yes
        sim: track

        sollwert:
            knx_dpt: 5001
            visu_acl: rw
            type: num
            enforce_updates: yes
            cache: True

        taster:
            knx_dpt: 1
            visu_acl: rw
            type: bool
            cache: True
            enforce_updates: True

And it could be implemented in other structs like this:

licht_dimmen:
    struct: dimmen
licht_dimmen_dual:
    warm:
        struct: dimmen
    kalt:
        struct: dimmen

[msinn]

@onkelandy

How good are the chances to have sub-structs available within a struct hierarchy?

Ist the functionality not working for you?

[onkelandy]

Currently the substruct has to be defined in the "root" item. So from my example at the end of the previous post the licht_dimmen currently works but putting the substruct into an additional item like warm or kalt doesn't work. At least it didn't after the last big update when we've discussed this.

[msinn]

Ok, i have to look into it to find why it is not working any more.

[thelsing]

Was anything done here? Last time I checked (11.11.20) nesting of structs didn't work in develop.

[onkelandy]

It works but only in a limited way as described in my post above. You can only put the nested struct in the "root" item. So this works in etc/struct.yaml:

substruct:
   item:
       type: foo

structexample:
   item1:
      type: bla
   struct: substruct

Hoping to have the option to implement substruct on other hierarchy levels soon, though, as this would help alot.

[thelsing]

Is there any plan to fix this? If not why not roll the struct handling back to the initial one an apply my patch above? I current struct handling code is a bit to complicated for my python foo.

[Morg42]

Should be fixed with @msinn's last updates in develop :)