search

wk275 / qbTools-v2

Tooling environment for Qbus and MQTT via Nodered.
GNU General Public License v3.0
2 stars 0 forks source link

Feature proposal for binary sensor and additional entity properties #8

Closed asulonn closed 8 months ago

asulonn commented 10 months ago

Feature proposal:

I have some PIR sensors and door/window contacts in my Qbus installation. I wanted to have them as binary inputs in HA. I also wanted to have them show as actual motion sensors and door contacts.

I changed your NodeRed code somehow, to accomplish this. Because Qbus makes no difference between an input or an output, and so does not provide this data through the API, the idea was to use the Qbus item name to find out if it is a sensor and what kind of sensor/output it is.

I made changes to the code in the "HA entity" function module in "QBUS HA" subflow and to the "Classes" function node in the "Global functions & classes" flow. I also created two environment variables in the "QBUS HA" subflow:

BinarySensorRegex: The idea is that when a Qbus item name matches the regex string in "BinarySensorRegex", it is made into a binary sensor. In my case for instance this regex string is: "SEN|PIR"

ExtraHaProperties: The ExtraHaProperties structure can provide a json struct that contains an array of objects with the properties: regex, haProperty and propertyValue. The idea is that for a Qbus item name that matches the "regex" pattern, the haProperty will be added for the item, and the value of that property will be set to the value of "propertyValue" In my case for intance this struct looked like this:

[
    {
        "regex": "MC",
        "haProperty": "device_class",
        "propertyValue": "door"
    },
    {
        "regex": "MC",
        "haProperty": "state_on",
        "propertyValue": false
    },
    {
        "regex": "MC",
        "haProperty": "state_off",
        "propertyValue": true
    },
    {
        "regex": "PIR",
        "haProperty": "device_class",
        "propertyValue": "motion"
    },
    {
        "regex": "ST_",
        "haProperty": "device_class",
        "propertyValue": "outlet"
    }
]

The structure above for instance sets the device_class for all items with "MC" in their name to "door" And inverts the value of the door sensors in HA.

The code change makes it easy to customise home assitant entities to personal needs. Feel free to do with it what you want!

Changed code in the files below: Classes.txt HA entity.txt

wk275 commented 10 months ago

Good idea.

Reviewed your code changes. Same structure same flavor, only the relevant parts touched so it is easy to compare. Great, I like it 👍 I will copy your code and tag the change with your name, in the next release if it is OK for you. Could you please provide an export of the "Qbus HA" node, so I have the complete solution.

One thing to do is checking how this code change impacts the "HA" node which is used for non QBUS HA entities. This node also uses the same class. Enabling this function here should be great. Thanks a lot

asulonn commented 10 months ago

The export for the QBUS HA sub-flow: (I undid the changes that i had made to the broker and to the position of some of the nodes that i moved around, so only the changes that matter for this feature should be left) QBUS HA.txt

One thing to do is checking how this code change impacts the "HA" node which is used for non QBUS HA entities. This node also uses the same class. Enabling this function here should be great.

I did not fully inspect the rest of the nodes deeply, but do you mean the subflow "HA"? I think the same code that was added to HA QBUS could be added to the "HA" node. This way MQ entities could also become binary switches and/or get extra properties. If you like i could look into this later this week.

Something i wasn't sure about: For the binary sensor in "I_setBinarySensor()" i had to use:

payload_on: this.I_createSwitchOnOff(attributes[i], "state")?.on,       //In case of a binary sensor, the payload needs to be like what the state is for a switch.
payload_off: this.I_createSwitchOnOff(attributes[i], "state")?.off,     //In case of a binary sensor, the payload needs to be like what the state is for a switch.

I used the 'state' as input for the function. If i used 'payload' as input, it did not work. What i could read from the HA docs is that i needed the values that were returned when i used "state'. Meaby you have more experience with the whole state-payload stuff....

wk275 commented 10 months ago

Thinking about how to implement your idea in node HA, I think that a few changes have to be made keeping in mind that the majority of nodes I implemented can be driven by node parameters (static) but also by message parameters (dynamic). E.G a Qbus Home assistant entity could also be created by the "HA" node iso the "QBUS HA" node by passing the correct QBUS node or message parameters. Node parameters have priority on message parameters: See export QbusFlow.txt for detail

For the binary_sensor implementation I think following changes have to be implemented in the HA node:

  1. adding an option to node parameter entity_type
  2. adding a node binary_sensor parms
  3. changing HA Create to get the binary_sensor parms

That should work. But option 2 bothers me. Up to now, I created node parameters for specific entity types. With the future in mind this is not ideal, because I learned that the structure of a node should change as less as possible between different releases. If the code behind it or the content of an existing parameter changes that is no problem. The idea behind it is that a change may not break previous implementations. So following changes should be made as well:

  1. change the HA node parameters to 1 parameter iso a parm per entity type. E.G entity_parms : {"entity_type": "sensor", "state_topic": "cloudapp/QBUSMQTTGW/UL1/UL176/state"}
  2. adapt HACreate to support this change
  3. get rid of the current entity type in the node and in HAcreate as it is now part of the entity_parms section. That should do the job :-) Feel free to implement it.

In class Ha, the general idea is to check if entity specific parameters are supplied or not. Today this is done by the I_checkHaParms method. The problem is that it is a general checking not specific related to an entity. That's why there are exclusion as well :-) The idea behind it is to support the user with defining the parameters and get at least the required parameters.

With this new enhancement I noticed that I should implement required, default and optional entity parameters. E.G.

You had a little bit of luck with the device_class and state_on, state_off parameters . They were already implemented. Try to specify icon: mdi:home-circle I'm 100% sure it will not work :-)

In the Qbus forum I noticed that some users complain about the complexity of HA integrations tools. Don't know if they were pointing at the qbTools solution or not. To me it seems not complex at all. Certainly not for the QBUS part, because there every parameter is specified out of the box and depends upon the UBIE MQTT implementation.

It becomes complex if you are trying to generate mqtt entities yourself, because then you'll need to know what HA MQTT entity parameters you should specify and how they should be constructed. So keeping this in mind, I'm searching for a solution that makes it less complex and maybe the required, optional and default parameters can help. :-)

wk275 commented 10 months ago

Something i wasn't sure about: For the binary sensor in "I_setBinarySensor()" i had to use: payload_on: this.I_createSwitchOnOff(attributes[i], "state")?.on, //In case of a binary sensor, the payload needs to be like what the state is for a switch. payload_off: this.I_createSwitchOnOff(attributes[i], "state")?.off, //In case of a binary sensor, the payload needs to be like what the state is for a switch. I used the 'state' as input for the function. If i used 'payload' as input, it did not work. What i could read from the HA docs is that i needed the values that were returned when i used "state'. Meaby you have more experience with the whole state-payload stuff....

In the HA doc you read

payload_off string (optional, default: OFF) The string that represents the off state. It will be compared to the message in the state_topic (see value_template for details)

payload_on string (optional, default: ON) The string that represents the on state. It will be compared to the message in the state_topic (see value_template for details)

It means that it will use the value of the state_topic and value_template to get to the actual value In de code this is translated by the state type.

The payload type expects to get a command type value. something like
{"id":"UL192","type":"state","properties":{"value":40}}

So it is 100% correct

wk275 commented 10 months ago

But ..... testing further revealed that it is weird if you want to override the payload_on and payloadoff parameters because in the node parameters you have to define state whereas in the documentation payload_ is specified

{ "regex": "MC", "haProperty": "state_off", "propertyValue": true },

The solution is to use payload in I_setBinarySensor:

payload_on: this.I_createSwitchOnOff(attributes[i], "payload")?.on,       //In case of a binary sensor, the payload needs to be like what the state is for a switch.
payload_off: this.I_createSwitchOnOff(attributes[i], "payload")?.off,     //In case of a binary sensor, the payload needs to be like what the state is for a switch.

and to specify in the I_setQbusDefaults case "binary_sensor":

this.qbusCollection.payload_on = true;                                //#E2515 a binary_sensor uses the state_topic and value_template to get the state value                               
this.qbusCollection.payload_off = false;                              //#E2515 a binary_sensor uses the state_topic and value_template to get the state value
asulonn commented 10 months ago

I will try comment on the rest later, but to start meaby about your last posts:

But ..... testing further revealed that it is weird if you want to override the payload_on and payloadoff parameters because in the node parameters you have to define state whereas in the documentation payload_ is specified

That is what i meant with my statement "Something i wasn't sure about". Because I had to use payload_on: this.I_createSwitchOnOff(attributes[i], "state")?.on, instead of payload_on: this.I_createSwitchOnOff(attributes[i], "payload")?.on, This did not seem right to me. I followed your instructions and changed the code in I_setBinarySensor and I_setQbusDefaults, and now it works as it should with:

{
"regex": "MC",
"haProperty": "payload_off",
"propertyValue": true
},

Great!! ;-)

asulonn commented 10 months ago

Ok, my response came a few days later. But i had to find a moment where i had some time to read your post and analyse what you wrote and ment :-)

hinking about how to implement your idea in node HA, I think that a few changes have to be made keeping in mind that the majority of nodes I implemented can be driven by node parameters (static) but also by message parameters (dynamic). E.G a Qbus Home assistant entity could also be created by the "HA" node iso the "QBUS HA" node by passing the correct QBUS node or message parameters. Node parameters have priority on message parameters: .... get rid of the current entity type in the node and in HAcreate as it is now part of the entity_parms section. That should do the job :-) Feel free to implement it.

I think i understand what you mean with the code above. As is already done for the other types => Do the same for the binary sensor. But instead of making params for each type => Make only one. Sounds good!

Some ideas: -What would you say about getting rid of all the environment variables of the HA node? I think they overcomplicate things and they are only used (for what i could see) in "Unit test - qbTools". I propose: HA-node receives a message containing with the parms (in a json) it needs. The parms-struct is handed to a function of the HA-class that turns it into an Home assitant config message. In the Unit test, the data that is now passed through the environment vars, could be passed through the injection-node or through some 'set-node' -Make the HA-class unaware of Qbus-entities. Qbus class gets "qbus params"-struct and converts it to a HA-params struct (the sam params struct as the one passed as input in the point above). This way each has its own responsibility. Qbus class= convert to something for HA class; HA-class = convert standard input struct to actual HA config json for sending to MQTT

Meaby these proposals do not make sense and are because of i did not fully understand/look at all of the code. Feel free to tell me so if that is the case! The thing i struggled with (especially in the beginning): parms parms parms :-) That name seem to be all over the place, coming from many different sources. I had a hard time to find the flow of the data/code. That's why i was thinking if it is possible to make it more sequential. With clear input and clear output

In class Ha, the general idea is to check if entity specific parameters are supplied or not. Today this is done by the I_checkHaParms method. The problem is that it is a general checking not specific related to an entity. That's why there are exclusion as well :-) The idea behind it is to support the user with defining the parameters and get at least the required parameters. With this new enhancement I noticed that I should implement required, default and optional entity parameters. E.G. a sensor entity requires only a state_topic, but you may optionally specify a device_class, json_topic, ... too a binary sensor entity requires a state_topic and a device_class and optionally a payload_on, payload_off. ...

This sounds good! Only check for required vars only per type. For instance: binary sensor is only state_topic and payload_on and payload_off (because default is "on" and "off" instead of true and false, otherwise state_topic would be enough)

You had a little bit of luck with the device_class and state_on, state_off parameters . They were already implemented. Try to specify icon: mdi:home-circle I'm 100% sure it will not work :-)

I know :-) i only saw afterwards you did the checking and that device_class was there!

In the Qbus forum I noticed that some users complain about the complexity of HA integrations tools. Don't know if they were pointing at the qbTools solution or not. To me it seems not complex at all. Certainly not for the QBUS part, because there every parameter is specified out of the box and depends upon the UBIE MQTT implementation.

"Not complex at all" That is a bold statement :-p I must admit that i had to spend some time before getting some idea about how it all works. I think the fact that it is made in node red has a lot to do with this. I think that you have done a magnificent job about working with these classes and having a callstack and error handling and all that. But still the fact that there is no cross-referencing, no global code search, that the code is spread/used over different nodes and that there is quite some code you had to add to have some decent debugging, make it less transparent... Not to take anything away from your accomplishment! I think it is great and i certainly would not have been able to do better. I could understand when you have no programming background this is hard to understand, but then again, when i first used it, i had no clue what was under the hood, i just implemented it and it worked out of the box, so no need to understand it all if you just want to use it...

It becomes complex if you are trying to generate mqtt entities yourself, because then you'll need to know what HA MQTT entity parameters you should specify and how they should be constructed. So keeping this in mind, I'm searching for a solution that makes it less complex and maybe the required, optional and default parameters can help. :-)

I understand and agree. Meaby this can be solved by adding help text for the HA node with some examples and info. When i find a new node i want to use, i usually check the help page for the possible input/message variables and their meaning. Some good information in that section could go a long way. The question also is how many people (that do not want to dive deep into this) want to use it for more than connecting their Qbus to their HA.... Like i mentioned earlier, only doing that was pretty straightforward to implement.

wk275 commented 9 months ago

Some ideas: -What would you say about getting rid of all the environment variables of the HA node? I think they overcomplicate things and they are only used (for what i could see) in "Unit test - qbTools". I propose: HA-node receives a message containing with the parms (in a json) it needs. The parms-struct is handed to a function of the HA-class that turns it into an Home assitant config message. In the Unit test, the data that is now passed through the environment vars, could be passed through the injection-node or through some 'set-node' -Make the HA-class unaware of Qbus-entities. Qbus class gets "qbus params"-struct and converts it to a HA-params struct (the sam params struct as the one passed as input in the point above). This way each has its own responsibility. Qbus class= convert to something for HA class; HA-class = convert standard input struct to actual HA config json for sending to MQTT

will be continued :-)

wk275 commented 9 months ago

The thing i struggled with (especially in the beginning): parms parms parms :-) That name seem to be all over the place, coming from many different sources. I had a hard time to find the flow of the data/code.

it's easy :-)

"Not complex at all" That is a bold statement :-p I must admit that i had to spend some time before getting some idea about how it all works. I think the fact that it is made in node red has a lot to do with this. I think that you have done a magnificent job about working with these classes and having a callstack and error handling and all that. But still the fact that there is no cross-referencing,

I meant that is is not complex to use it out of the box for creating HA entities for QBUS :-) It becomes indeed a different story if you want to change the code. I use Visual code to develop the code. It comes with some functions like comparing two versions, code referencing etc. What is really missing is a development environment for node red subflows. At least I did not find one. That's why the code has the trace & debug facilities :-)

-What would you say about getting rid of all the environment variables of the HA node? I think they overcomplicate things and they are only used (for what i could see) in "Unit test - qbTools"

I use the HA node also for creating non-QBUS HA entities. I've some Shelly devices which are all MQTT connected:

The node "QBUS - SHELLY" integrates some of these Shelly devices with QBUS and uses HA node too :-)

So I rather keep the parameters in the HA-node, but in stead of having an environment parameter section for every entity type I will combine them in 1 parameter section for all.

wk275 commented 9 months ago

Last thing to resolve for now :-) How to implement specific default, required and optional parameters. I think this boils down to node specific parameters. I was thinking about the following structure for the HA node. The object value is the default for optional parameters and an example for required parameters. 

{
    "climate":
    {
        "optional":
        {
            "unique_id": "virtual_test_HVAC",
            "action_payload": "heating",
            "current_temperature_payload": 25,
            "modes": ["heat", "off"],
            "mode_command_payload": "off",
            "mode_state_payload": "off",
            "precision": 0.1,
            "preset_mode_command_payload": "MANUEEL",
            "preset_mode_state_payload": "MANUEEL",
            "preset_modes": ["MANUEEL", "VORST", "ECONOMY", "COMFORT", "NACHT"],
            "temperature_command_payload": 30,
            "temperature_state_payload": 30,
            "temperature_unit": "C",
            "temp_step": 0.5
        },
        "required":
        {
            "name": "virtual test HVAC",
            "action_topic": "virtual/test/HVAC/action",
            "current_temperature_topic": "shellies/shellyht/sensor/temperature",
            "mode_command_topic": "virtual/test/HVAC/mode",
            "mode_state_topic": "virtual/test/HVAC/mode",
            "preset_mode_command_topic": "virtual/test/HVAC/preset",
            "preset_mode_state_topic": "virtual/test/HVAC/preset",
            "temperature_command_topic": "virtual/test/HVAC/temperature",
            "temperature_state_topic": "virtual/test/HVAC/temperature"
        }
    }
}

To implement it I was also thinking about a separated node or subflow to publish to the global.context. This global.context is easily accessible in class and node code. The alternative is to write it in a node red module which ask the right questions dynamically. This seems to be more user friendly as it will guide you through the possible options. Downside: I have less experience with developing modules :-)

asulonn commented 9 months ago

I think that the general parameters should stay as environment parameters in HA. But more advanced parameters should go in 1 parms environment section. In my opinion you should be able to specify msg.parms and env.parms. So one could choose between the static and the dynamic parms. Or a combination of both :-)

Ok fair enough, this probably does not affect the "class-code" that much anyway. Only the code that is in the function node.

In previous releases I had a qbus-ha class that was used to generated the HA entities. But again, than you need to adapt changes in two places i.s.o. one. That is why I converted it to one class. Creating HA entities should not be part of the qbus class imo. The qbus class handles a qbus message. Again splitting it up in a Qbus-HA class which enherits from Qbus seems a better solution to me. And than I 100% agree with your idea :-) The qbus-ha node should than incorporate a HA-node and passing all parms via a msg.

What i meant, is that there seems to be a lot of "qbus" in the HA-class. For example the image below shows the HA constructor, which starts with Qbus statements. I would think that the Qbus-class would take care of this. image

The flow in my mind would be: Qbus entities: Env data \ "qbusInputdata" >Qbus class> "haInputData" >HA class> "mqttConfig" >MQTT Msg data/

Other entities: Envir data \ "haInputData" >HA class> "mqttConfig" >MQTT Qbus data/

This seems a logic flow, but i can imagine that once writing the actual code/knowing more about the specific needs, that initial idea of the flow changes. And things are less black and white than they were in the beginning :-)

It becomes indeed a different story if you want to change the code. I use Visual code to develop the code. It comes with some functions like comparing two versions, code referencing etc. What is really missing is a development environment for node red subflows. At least I did not find one. That's why the code has the trace & debug facilities :-)

Ok yeah i should try looking at it in VS code. That would be easier! Node red is kinda double. On one hand it enables you to communicate with a lot of third party stuff for which you would/could not want to develop code on your own and it makes it easy to do some really cool stuff through the ui. But when you want something more advanced and/or you're used to write code in all text, the ui/message flow is quite a nuisance ... Pretty nifty how you found ways to make it all work like that!

To implement it I was also thinking about a separated node or subflow to publish to the global.context. This global.context is easily accessible in class and node code. The alternative is to write it in a node red module which ask the right questions dynamically. This seems to be more user friendly as it will guide you through the possible options. Downside: I have less experience with developing modules :-)

I'm not sure what you mean by that last bit, but it sounds like a good plan to separate the validation structure with the variables it should/could hold in a separate spot. This way it is clean and easy to check and update these variables. The validate method meaby could be a generic method which is handed the HAparams and the struct that is applicable for the given entity-type. If i understand you would make the HA-class more entity-type oriented? Now there is is the"set" method per entity-type. This could be done the same for the "validate" method (or make it generic as described above.

If you want i would be willing to take a stab at the code and try some ideas, but it's quite busy atm, so i'm not sure about planning 🙈 I would also understand if you rather keep more control over this code yourself :-)

wk275 commented 9 months ago

accidentally closed the issue I reopened it :-)

wk275 commented 9 months ago

If you want i would be willing to take a stab at the code and try some ideas, but it's quite busy atm, so i'm not sure about planning 🙈 I would also understand if you rather keep more control over this code yourself :-)

Must think about it. Writing code with 2 is more difficult imo. To make it work you need a kind of structure to do that. E.G split the code in specific parts per entity? Have some automatic testing against it, etc. Git branches, ... Too difficult at this time I think.

I learned that you understand the code very well and have valuable input . So you can always make suggestions, ideas or provide me with code as you did before :-) It is very much appreciated.

I made a little bit of progress in checking the input parameters. Was looking at following structure to hold the information: 

"sensor": {
        "required":
        {
            "name": "@unique_id",        // @ refers to another parameter. So if name is not defined, it will take the value of unique_id
            "unique_id": "@name",
            "object_id": "@unique_id",
            "state_topic": ""
        },
        "optional":
            [
                "device_class",
                "enabled_by_default",
                "expire_after",
                "force_update",
                "icon",
                "json_attributes_template",
                "json_attributes_topic",
                "suggested_display_precision",
                "state_class",
                "unit_of_measurement",
                "value_template"
            ],
        "qbTools":
            [                               // special qbTools parameters
                "state_attributes",         // only these attributes will be processed
                "state_payload"             // structure auto defines value_template 
            ],
        "examples":
            [
                {
                    parms: {
                        "name": "test1_shellymotion",
                        "state_topic": "shellies/shellymotion/status",
                        "state_payload": {
                            "tmp.value": 20
                        },
                        "icon": "mdi:thermometer"
                    },
                    "example shellymotion mqtt msg": { "motion": false, "timestamp": 1695458711, "active": true, "vibration": false, "lux": 0, "bat": 52, "tmp": { "value": 21.8, "units": "C", "is_valid": true } },
                }
            ]
    }
asulonn commented 9 months ago

I guess i didn't respond yet :-)

Must think about it. Writing code with 2 is more difficult imo. To make it work you need a kind of structure to do that. E.G split the code in specific parts per entity? Have some automatic testing against it, etc. Git branches, ... Too difficult at this time I think.

Yeah you are correct! Coworking on code makes stuff more complicated. Just offering in case you wanted help :-) What i was thinking, would it be an idea to put up the NodeRed json/folder also separate in the repo? This would be nice for updating. Now i download the zip file from the repo, Copy the NodeRed folder to the docker place, spin up the NodeRedQB container and copy and paste the json to my own environment. This is not such a big deal though, so if it is a lot of work/hassle it is fine the way it is :-)

I learned that you understand the code very well and have valuable input . So you can always make suggestions, ideas or provide me with code as you did before :-) It is very much appreciated.

Ok! Very well. If you need to have some code tested, i'm always willing to help out!

"sensor": {
        "required":
        {
            "name": "@unique_id",        // @ refers to another parameter. So if name is not defined, it will take the value of unique_id
            "unique_id": "@name",
            "object_id": "@unique_id",
            "state_topic": ""
        },
       .....

Looks good! I'm curious now how it will turn out :-D

wk275 commented 9 months ago

Below the first beta of the new HA implementation. It uses a "HAparms" node in the flow "Global functions & classes" to build the HA entities. Nodes HA and QBUS HA were also changed. I added a flow "Unit test - parmsStructure implementation" to get an idea of the possibilities. You have to change the parameters in some places to match your setup.

Files to import in node-red: Global functions & classes.txt HA.txt QBUS HA.txt Unit test - parmsStructure implementation .txt

The HAparms node contains the following sections

@value strings in the entity and class sections refer to another parms.parameter, entity.parameter or class.parameter, ' e.g. "name": "@unique_id" means: if name parameter is not defined, it will get the value of the unique_id parameter' values without an @ in the entity and class sections refer to a default value", ' e.g. "name": "one" means: if name parameter is not defined, it will get the default value "one"

Not all possible entity parameters are specified. E.G. If you want to use the availability parameters just add them in the required or the optional section. Carefully define optional defaults, because they will always appear in the entity definition, needed or not :-).

Asulonn, If possible could you also test the code with your setup? It will probably reveal some bugs in the new code.

Txs a lot.

asulonn commented 9 months ago

Ok, so i have been testing. At first sight it all looks good, I took the config from what i had before and put it in "HAparms":


"regexPre": [
            {
                "name_regex": "SEN|PIR",
                "attributes":
                {
                    "entity_type": "binary_sensor",
                },
            },
        ],
        "regexPost": [
            {
                "name_regex": "MC",
                "attributes":
                    { 
                        "device_class": "door",
                        "payload_on": false,
                        "payload_off": true
                    }
            },
            {
                "name_regex": "PIR",
                "attributes":
                    { "device_class": "motion" }
            },
            {
                "name_regex": "ST_",
                "attributes":
                    { "device_class": "outlet" }
            }

That all seems to work like it used to! The only thing that is strange, is that now all the switches have ".value" added to the end of their name. "LMP_Buiten_Voordeur" now is "LMP_Buiten_Voordeur.value" ("LMP_Buiten_Voordeur_value" is new the topic) The other types don't seem to have this "value" added to them. I have been looking a bit in your code, but i have not yet found out why. I don't think this is something you would want?.. Do you have any idea?

I must say i haven't been able to look at the whole new code in depth. But from a glance it looks very good! I like how you separated the "config" (optional & required) values from the code! Great work!

wk275 commented 9 months ago

You are wright. It is not so elegant :-) This is due to the change in the Ha class from a specific approach (e.g; setSwitch, setSensor, ... to a generic one.

The .value is the attribute used in the Qbus payload syntax. class.attributes are used to loop through the payload object. e.g. if you create a sensor for a thermo/climate entity the attributes "currRegime", "currTemp", "setTemp" are added to the name, unique_id and object_id. In this case it make sense. For a switch it is not needed. I agree.

Changed the class to reflect this change.

Classes.txt

Txs for testing.

asulonn commented 9 months ago

I tested your changes. They work! No more ".value". It seems like all entities are like they were before. Very Nice!

I was checking out what i could do with the "Device name". In the prev version i tweaked (rolled back to the prevprev version) your code so the device name kept being like "Q light" or "Q switch" for instance, but in the new version all devices are named "Q". I liked how it was before, so i was checking out if i could do this through the post processing like this:

"regexPost": [
{
    "name_regex": ".*",
    "attributes":
        { "device.name": "@device.identifiers" }
},

Because i noticed the "device.identiefier" seemed to be composed the way i would like, i thought i could use it and it would be translated because it had the "@" in front of it. But it just copied it literally. I must have misunderstood your explanation above! I have not checked out how this works, meaby it is not the intention to work like that 🙈

When i deploy i noticed that there are a 4 errors image

The nodePath id's translate to: "Weather integration"/"HA"/"HACreate". I'll have to look into what happens there exactly and/or if something went wrong in updating the nodes to the new version.

Just thinking out loud: i like how you put the "regexPost" and "regexPre" in the function node. As hinted before, i like all text :-). I was just wondering if this might be a little intimidating to less experienced users. Even though i don't have a straight alternative to make it easier... Anyway, it is optional, so no real necessity to use by less experienced users... :-D

Txs for testing.

No worries! My effort is very little compared to yours ;-) Very nice of you to share the whole project by the way!!

wk275 commented 9 months ago

The @redirect was never intended to be used in the regex parms. So it will not work there. But it can be developed if you want :-) The description is indeed not so clear, I'll change it.

They do have an effect in the entity and class parameters where they behave as a link to an already defined property.

However, the logic to create default device parameters was not OK and has been corrected in the "classes 2.5.3.txt" file below. Change the default in the optional parm sections from "device.name": "@device.name", to "device.name": "@device.identifiers", and it should work.

Although this is a default on class code level, a parameter defined on HA node level or in a msg.parms like "device.name": "test" will still overwrite the default HAparms as it should be imo..

Be aware that the naming conventions changed in Homeassistant 2023/08. They do take the device name into account when creating a default friendly name. To shorten the name, the "Q sensor" was replaced by "Q" :-)

Classes 2.5.3.txt

Success

asulonn commented 9 months ago

The @reDIrect was never intended to be used in the regex parms. So it will not work there. But it can be developed if you want :-) The description is indeed not so clear, I'll change it.

Don't feel pressured by me! Go as far as you think you should/want to go ;-) Because if you make this possible, the next thing i will ask you is to be able to do stuff like { "device.name": "Q [entity_type]" } (make combo's of vars and literal text) :-D They are just ideas that i think might be nice, but meaby also might be to far fetched and/or make everything more complicated than needed...

However, the logic to create default device parameters was not OK and has been corrected in the "classes 2.5.3.txt" file below. Change the default in the optional parm sections from "device.name": "@device.name", to "device.name": "@device.identifiers", and it should work.

Ok i understand, but it feels like the optional and required vars is more "the standard" and all custom/user specific stuff should be in the post and pre processing. Otherwise, when you push an update, and "we" copy it, tweaks and changes might get lost. Like how i changed the device naming in my instance of the code before, but with the update it is back to the standard... That being said: i tested your new code "Classes 2.5.3.txt" and it works (i did not notice any change). I also tested changing "device.name": "@device.name", to "device.name": "@device.identifiers" and that also works (tested with the new code, i didn't test with the old code) => The name is back to how i had it before.

Although this is a default on class code level, a parameter defined on HA node level or in a msg.parms like "device.name": "test" will still overwrite the default HAparms as it should be imo..

I'm not totally sure what you mean, but i think i agree :-D Default values are just default values. If any other source of info for the value, it is overwritten with that other info.

Be aware that the naming conventions changed in Homeassistant 2023/08. They do take the device name into account when creating a default friendly name. To shorten the name, the "Q sensor" was replaced by "Q" :-)

I understand. And i don't mind the slightly longer name so much. But the different types of entities still fall under different devices in HA. All those devices have the same name now. In some cases in HA where you have to select a device first and an entity second, this is anoying: image That's why i like the name with entity better.

When i deploy i noticed that there are a 4 errors -Picture- The nodePath id's translate to: "Weather integration"/"HA"/"HACreate". I'll have to look into what happens there exactly and/or if something went wrong in updating the nodes to the new version.

I checked some more, the errors seem to be triggered when the environment pars are as shown below: image On the left you can see where i put a "warning" in the code of the "HACreate"-function node, and on the right you can see the actual warning with the content of the "envParms" and following the error that is triggered in the class-code. I can investigate further if you want, but meaby you see instantly what is wrong :-)

wk275 commented 9 months ago

This error occurs if you did not pass an entity_type in the msg.parms or in the node parameters. If you look at this.parms it indeed shows no entity_type definition.

This is because the node HA parms was changed in your version I deleted the name and entity_type properties. In the HA node export below this is corrected :-) HA.json

asulonn commented 9 months ago

This is because the node HA parms was changed in your version I deleted the name and entity_type properties. In the HA node export below this is corrected :-) HA.json

No more error errors after updating with "HA.json"! Great work!

wk275 commented 9 months ago

Don't feel pressured by me! Go as far as you think you should/want to go ;-) Because if you make this possible, the next thing i will ask you is to be able to do stuff like { "device.name": "Q [entity_type]" } (make combo's of vars and literal text) :-D They are just ideas that i think might be nice, but meaby also might be to far fetched and/or make everything more complicated than needed...

It was challenging :-)....... but I developed your feature as I needed it also for some MQTT topics. There is a little change in the HAparms definitions due to properties that can contain a [ ... ] like value_template. The redirect is now detected by a [@........] definition and can be used everywhere inHAparms and in the node or message parms. E.g. "name" : "[@unique_id]

I also added an extra #slice functionality that I needed for some MQTT devices and yes additional literal text is also possible :-)

slice has parameters.

1) slice separator character 2) slice join character 3) start offset, end offset (keep in mind that these are offsets so they start with 0 ==> 1,4 = copy offset 1, 2 and 3 In fact this is a code combination: redirectTargetValue =redirectTargetValue.split(sliceSeparator).slice(sliceStart, sliceEnd).join(sliceJoiner); E.g. "topic": "shellies/shellyplug2/relay" "name": "test[@topic#slice(/,,1,3)] after redirection and slice the result is "name": "test_shellyplug2_relay"

Below the HAparms and classes files

HAparms.txt Classes 2.5.6.txt

asulonn if possible could you test the code?

Btw. this is also possible :-) "name": "test_[@topic#slice(/,_,1,3)][entity_type]

asulonn commented 9 months ago

Looks very cool!

asulonn if possible could you test the code?

Yes! I want to test this for sure! Possibilities seem very flexible! Exciting! 😀

It will be something for the weekend though! Busy week for work!

asulonn commented 8 months ago

Ok, so i did some testing! Very impressive work man! Very cool how you did it! And on such short notice :-o

This is what i added to my adaptations in the post regex:

{
    "name_regex": ".*_.*",
    "attributes":
        { "device.name": "QB [@entity_type]" }
}

This works great! At first i used ".*" but then it changed the MQTT objects also, and i only wanted the QB ones :-) Very nice! The other pre/post regex changes i already had (discussed earlier in this thread, also all still work ;-) )

I must say i did not yet try the slicing! I did not yet have a real use case :-)

What i did try, but could not get to work, is change the device type of the "weather switches": I'm using the weather data nodes that you made and they result in MQTT topics like: "weather/switch/nord" They show up as switches in HA. I wanted to make them binary sensors. I was wondering on which part/value the regex searches for the pre regex? I figured it would be good to look for the name of the switch with no letters in front or behind to prevent false positives, but that does not seem to work. What i added to do this to the pre-regex:

{
    "name_regex": "\bnord\b",
    "attributes":
        { "entity_type": "binary_sensor"}
}
wk275 commented 8 months ago

In the regexPre I take into account the basename. If you look in HA > setup> devices> entitities and search for "nord" you'll see a name like "M weather.nord". This name is constructed as: "" + " "+ "basename" + "." + attribute. So regexPre works on weather. Nothing else. As a matter of fact this is normally the msg.topic. Attributes are defined in the msg.payload.

The regexPost searches in the constructed name which is "weather.nord".

For qbus the basename is always the qbus output name. For thermo devices the attributes are currRegime, currTemp and setTemp. The best way is to see what the name looks like in the HA overview.

1st solution:

In the HA node with entity_type switch, change it to binary_sensor. In the HA overview, you'll see now weather.nord 2 times. First as a swith, secondly as a binary sensor. If you do not want the switches, delete them in mqtt topic homeassistant/switch/mqtt

Then if you want to add a device_class which is the same for all the binary_sensors. Add it in the HA node in the parms section

{
"device_class": "????"
}

If you want a device class per binary_sensor you can define it in the regex_post object like;

{
                "name_regex": "^weather.humidity$",
                "attributes":
                    { "device_class": "moisture" }
},

2nd solution is to define a specific "MQTT get" node per switch.

E.g. for the humidity switch change the "MQTT get" node name to weather/switch/humidity, change in the "HA node" the entity type to "binary_sensor" and in the parms section you define the device_class as

{
                "device_class": "moisture" }
},

Copy and adapt the solution for the other switches.

This solution is more transparent, because all definitions are in one place.

3th solution:

Drop an inject node and a HA node on your canvas an connect them. Define in the HA node the name as weather_test.humidity and the entity_type as "binary_sensor". Deploy and execute. You'll get an error: Error: parameters (state_topic) missing in weather_test.humidity Define in the "HA node" parms section:

{ 
"state_topic": "weather/switch/humidity"
}

Error resolved and the binary switch is created with name "weather_test.humidity".

Finishing touch :-)

{
    "state_topic": "weather/switch/humidity",
    "device_class": "moisture"
}

4th solution:

Drop an inject node and a HA node on your canvas an connect them. Do not fill in name or entity_type. In the parms section use 

{
    "name": "weather_test2",
    "entity_type": "binary_sensor",
    "state_topic": "weather/switch",
    "payload": {
        "nord": "off",
        "humidity": "off"
    }
}

4 solutions, feel free to choose :-)

wk275 commented 8 months ago

Have just released the new version qbTools-2.5.7

asulonn commented 8 months ago

Hey man! Sorry for my late response! Work has been crazy last two weeks. Taking up most of free time to tinker! At last today i had some spare time to look at it! Thanks for your lengthy answer! I just saw it today you made such a long post :-|

4 solutions, feel free to choose :-)

However they looked like plausible solutions, I chose not to choose any of the 4 answers :-D

I was thinking, why can't i search any other properties to find out whether i want to change options... So i introduced a new property in the regexPre-struct => "propertyToSearch". In it you can indicate which prop you want to search with the regex string. If this property is not present or empty, it will just work like before.

All the changes i made are in "Classes" in "Global functions & classes". Below is the part of the code in which i made changes:

....
                    // ***************************************************************************************************************************************
                    // pre process regex parms - add/change specific parameters according to baseName (everything in the HA entityname before the first dot)
                    // ***************************************************************************************************************************************
                    let regexParms = parmStructure["ha"]["regexPre"];
                    let origName = this.assignObject(this?.parms?.name, this?.parms?.unique_id, this?.qbusCollection?.name);

                    //DEBUG. REMOVE
                    //if (this.parms?.topic && this.parms.topic.includes("weather"))
                    //{
                    //    node.warn(this.createFlatObject({ obj: this, propertyName: "" }));
                    //}
                    //DEBUG. REMOVE END

                    for (let i = 0; i < regexParms.length; i++) {

                        let propToSearch = origName;

                        if (this.isDefined({ obj: regexParms[i] })) {
                            //Check if the name for the property to look for is specified => If not use the 'name' of the entity by default. 
                            if (regexParms[i].hasOwnProperty("propertyToSearch") && regexParms[i].propertyToSearch) {

                                //Check if parms holds a property with the given name
                                if (this?.parms?.hasOwnProperty(regexParms[i].propertyToSearch)) {                     
                                    propToSearch = this.parms[regexParms[i].propertyToSearch]; //Use the value of this property to look in, instead of the 'name'
                                    //node.warn("Parms Property: '" + propToSearch + "' Value: '" + this.parms[regexParms[i].propertyToSearch] + "' Orig name: '" + origName + "'")    
                                }
                                //Check if qbuscoll holds a property with the given name
                                else if (this?.qbusCollection?.hasOwnProperty(regexParms[i].propertyToSearch)) {
                                    propToSearch = this.parms[regexParms[i].propertyToSearch];
                                    //node.warn("QB Property: '" + propToSearch + "' Value: '" + this.parms[regexParms[i].propertyToSearch] + "' Orig name: '" + origName + "'") 
                                }
                            }

                            if (propToSearch) {
                                let regex = new RegExp(regexParms[i].name_regex, 'g');
                                if (this.isDefined({ obj: propToSearch }) && regex.test(propToSearch)) {
                                    this.mergeObjects(this.parms, regexParms[i].attributes);
                                    this.I_debugTrace("Ha", "I_constructor-Pre-Regex-changesAfterMerge", { "this.parms.entity_type": this.parms.entity_type, regex: regex, regexParms: regexParms, "this.parms": this.parms });
                                }
                            }
                        }
                    }
                    this.I_debugTrace("Ha", "I_constructor-ParmsRegex", { "this.parms.entity_type": this.parms.entity_type, regexParms: regexParms, "this.parms": this.parms });

                    // ******************************
                    // check if entity_type is known
....

Below is an example of the regexPre struct:

"regexPre": [
            {
                "name_regex": "SEN|PIR",
                "attributes":
                {
                    "entity_type": "binary_sensor",
                },
            },
            {
                "propertyToSearch" : "topic",
                "name_regex": "weather/switch",
                "attributes":
                {
                    "entity_type": "binary_sensor"
                }
            },
            {
                "propertyToSearch" : "topic",
                "name_regex": "weather",
                "attributes":
                {
                    "device.name": "MQ [@entity_type]"
                }
            },
        ],

I changed this only for the regexPre, the same could be done for the regexPost...

As i mentioned before, these are just ideas about stuff that could be nice (for me) and you should by no means feel pressured to implement these if you think they are going to far and/or are cluttering the code/flow/user-friendliness! It's kind of spielerei :-) Because it really is not that important whether those weather entities are switches or sensors :-D

Have just released the new version qbTools-2.5.7

Should i update? Or am i pretty much up to speed with all the testing code you already gave me? :-)

wk275 commented 8 months ago

Asulonn,

Should i update? Or am i pretty much up to speed with all the testing code you already gave me? :-)

Do not know which is your original installation version. If it is 2.4.6 there were also some change on qbTools directory level. All changes are documented in the Release notes_qbTools-2.5.7 :-)

About the propertyToSearch change I need some time to investigate :-)