gw_evohome_rf

[martynwendon]

Hey there,

I'm taking a look at the new experimental build and have it up and running with the attached schema and config file.

schema.txt

evogateway.txt

I modified evogateway.py slightly just to give an additional file output with full debug logging so I can tail it alongside the console output.

Since I supplied a schema file, do I also need to create a devices.json file too? I assumed not, just wondered if it was better to create one as well or if it gave anything extra by doing so.

[smar000]

Hi

At this time, the devices.json is still ideally required even if you have the schema file as it is used for zone details etc. I am considering whether the app should standardise on just using the schema as that is what evohome_rf uses. However, this is yet another breaking change and for the purposes of this app, may not give significant additional benefit. Maybe worth looking at some way of having both devices.json and the schema live together nicely, with one overriding the other if both present.

Do you have any views?

[martynwendon]

Hey there,

I think for backwards compatibility it should perhaps pickup and use devices.json if it exists?

For new users such as myself though I think probably the schema.json is enough, it seems to be working fine as-is with that and picking up the names from the allowlist, at least in the log files.

2021-03-18 09:17:55,593 [346] || Bedroom Three Thermostat |                    |  I | temperature      | 00070D   || {'other_idx': '00', 'temperature': 18.05}
2021-03-18 09:17:55,595 [366]         -> process_gwy_message: 1. type(item): <class 'dict'>, item: {'other_idx': '00', 'temperature': 18.05} 
2021-03-18 09:17:55,597 [504]         -> mqtt_publish: 0. updated_payload: [{'other_idx': '00', 'temperature': 18.05}], type(updated_payload): <class 'list'>, new_key: None
2021-03-18 09:17:55,597 [505]         -> mqtt_publish:    payload: {'other_idx': '00', 'temperature': 18.05}
2021-03-18 09:17:55,598 [509]         -> mqtt_publish: 1. payload_item: {'other_idx': '00', 'temperature': 18.05}, type: <class 'dict'>
2021-03-18 09:17:55,599 [510]         -> mqtt_publish:    updated_payload: [{'other_idx': '00', 'temperature': 18.05}]
2021-03-18 09:17:55,600 [515]         -> mqtt_publish: 2. Posted subtopic: /evohome/gateway/_zone_independent/thm_22:046149/temperature/other_idx, value: 00
2021-03-18 09:17:55,602 [515]         -> mqtt_publish: 2. Posted subtopic: /evohome/gateway/_zone_independent/thm_22:046149/temperature/temperature, value: 18.05
2021-03-18 09:18:01,450 [345] 
2021-03-18 09:18:01,450 [346] || Internet Gateway   | NUL:262143         | RQ | actuator_state   | 00       || {}
2021-03-18 09:18:01,451 [366]         -> process_gwy_message: 1. type(item): <class 'dict'>, item: {} 
2021-03-18 09:18:01,452 [504]         -> mqtt_publish: 0. updated_payload: {}, type(updated_payload): <class 'dict'>, new_key: None
2021-03-18 09:18:01,453 [505]         -> mqtt_publish:    payload: {}
2021-03-18 09:18:03,452 [345] 
2021-03-18 09:18:03,453 [346] || Internet Gateway   | NUL:262143         | RQ | actuator_state   | 00       || {}
2021-03-18 09:18:03,457 [366]         -> process_gwy_message: 1. type(item): <class 'dict'>, item: {} 
2021-03-18 09:18:03,458 [504]         -> mqtt_publish: 0. updated_payload: {}, type(updated_payload): <class 'dict'>, new_key: None
2021-03-18 09:18:03,459 [505]         -> mqtt_publish:    payload: {}
2021-03-18 09:18:06,649 [345] 
2021-03-18 09:18:06,650 [346] || Internet Gateway   | Downstairs Hall Thermostat | RQ | schedule_sync    | 00       || {}
2021-03-18 09:18:06,651 [366]         -> process_gwy_message: 1. type(item): <class 'dict'>, item: {} 
2021-03-18 09:18:06,651 [504]         -> mqtt_publish: 0. updated_payload: {}, type(updated_payload): <class 'dict'>, new_key: None
2021-03-18 09:18:06,652 [505]         -> mqtt_publish:    payload: {}
2021-03-18 09:18:06,662 [345] 
2021-03-18 09:18:06,662 [346] || Downstairs Hall Thermostat | Internet Gateway   | RP | schedule_sync    | 00050... || {'change_counter': 2231, '_header': '0005'}
2021-03-18 09:18:06,663 [366]         -> process_gwy_message: 1. type(item): <class 'dict'>, item: {'change_counter': 2231, '_header': '0005'} 
2021-03-18 09:18:06,665 [504]         -> mqtt_publish: 0. updated_payload: [{'change_counter': 2231, '_header': '0005'}], type(updated_payload): <class 'list'>, new_key: None
2021-03-18 09:18:06,665 [505]         -> mqtt_publish:    payload: {'change_counter': 2231, '_header': '0005'}
2021-03-18 09:18:06,665 [509]         -> mqtt_publish: 1. payload_item: {'change_counter': 2231, '_header': '0005'}, type: <class 'dict'>
2021-03-18 09:18:06,666 [510]         -> mqtt_publish:    updated_payload: [{'change_counter': 2231, '_header': '0005'}]
2021-03-18 09:18:06,666 [515]         -> mqtt_publish: 2. Posted subtopic: /evohome/gateway/_zone_independent/ctl_controller/schedule_sync/change_counter, value: 2231
2021-03-18 09:18:06,667 [515]         -> mqtt_publish: 2. Posted subtopic: /evohome/gateway/_zone_independent/ctl_controller/schedule_sync/_header, value: 0005
2021-03-18 09:18:06,720 [345] 
2021-03-18 09:18:06,720 [346] || Internet Gateway   | NUL:262143         | RQ | actuator_state   | 00       || {}
2021-03-18 09:18:06,721 [366]         -> process_gwy_message: 1. type(item): <class 'dict'>, item: {} 
2021-03-18 09:18:06,722 [504]         -> mqtt_publish: 0. updated_payload: {}, type(updated_payload): <class 'dict'>, new_key: None
2021-03-18 09:18:06,723 [505]         -> mqtt_publish:    payload: {}

Does having the devices.json change the topic structure? I wasn't sure if the friendly names should appear in the MQTT topics?

For my purposes it doesn't matter since I'm using node-red on the other side, so all I do there is map topics / payloads into my existing schema.

I'm leaving this running at the moment alongside my existing very old Domoticz setup, I'm lucky to have a spare HGI80 to run in parallel.

For reference I've attached the topics I've collected so far.

One thing I did notice is that for the evohome controller there's so far no temperature or setpoint coming through. The evohome controller is used for the downstairs hall zone as the sensor, whereas all other zones use DTS92 room thermostats. I use the schema.json to assign it to the right zone based on comments at https://github.com/zxdavb/evohome_rf/issues/23.

I need to dig a little deeper into that as I'm not sure if that's something that's not coming across from the evohome_rf library or something getting missed in gw_evohome_rf. Or it could be that it's just not been running long enough for me to collect all the topics.

evohome_topics.txt

[smar000]

Thanks. Friendly names should ordinarily appear in the mqtt topics (makes it easier for me to see what is going on!).

WRT your temperature issue, I don't think that is related to either @zxdavb's evohome_rf library, nor the evogateway app, as I am getting the data through fine on mine (I also use the controller for temperature sensing in one zone).

[martynwendon]

Friendly names should ordinarily appear in the mqtt topics

Is that only when a devices.json is present? Or should they appear from just the schema.json allowlist?

as I am getting the data through fine on mine (I also use the controller for temperature sensing in one zone).

I get the data for the zone no problem (the hall zone), just no temperature / setpoint mqtt topic for the controller object under _zone_independent/ctl_controller.

[zxdavb]

I'm not sure if the following is clear, so I thought I'd add my 2c here...

Obviously, there is a difference between a device temperature, and a zone temperature.

A device's location in the schema is tenuous, and the schema is what the controller believes to be true. Of course, your own 'truth' (e.g. a TRV is in a zone) is irrelevant to the controller (as many people see when they first start using evohome_rf). The controller uses the schema to maintain state, and evohome_rf hopes to construct the same schema as the controller for reasons that will become obvious soon...

Many devices, including the controller have a built-in temperature sensor, and evohome_rf eavesdrops the reporting of these (which are cast for the controller's benefit), as it maintains the system state in a manner similar to the controller. Note the evohome controller doesn't have to broadcast it's own temperature to itself and, sadly, doesn't do so.

Now a zone's temperature (as reported by the controller) is the temperature of the device tagged in the controllers schema as the sensor for that zone. There is no other plan, such as:

• no alternative sensor device should the first stop reporting temps for whatever reason

• no average of all the temps of all the sensors in that zone

  evhome_rf uses discovery (i.e. it simply asks the controller) to get the schema - and thus to get teh device ID of each zone's sensor. Unfortunately, the controller doesn't tell you what zone it is the sensor for - that zone will simply be reported as having no sensor.

  There are other issues too, but the main point is this - evohome_rf cannot guarantee it knows what zone the controller is a sensor for (from discovery), although it could in many circumstances be very confident of which (from eavesdropping - this feature is not enabled, presently).

TL;DR: The controller uses device data (e.g. sensor temperatures, actuator heat demands), resolved against an internal schema, to determine the system state (e.g. zone temperature).

Some of this data (e.g. the controller's sensor temperature, each zone's heat demand) is not able to be determined by evohome_rf without behaving as the controller does, by resolving these packets against the schema.

[martynwendon]

Thanks for the explanation, I think it matches my understanding.

I've used the schema.json to allocate the controller as the sensor for zone 0 (and of course that is the "truth" as in it is bound to the system as the sensor for that zone).

So all the "zone" data is correct in that respect.

Independently of the zone data I also get temperature, setpoints, etc for each individual device (TRVs, room thermostats),

Note the evohome controller doesn't have to broadcast it's own temperature to itself and, sadly, doesn't do so.

From your comment here I think you're therefore saying that it's not possible to show the temperature / setpoint of the controller device itself as it's not physically sending that over RF?

Now a zone's temperature (as reported by the controller) is the temperature of the device tagged in the controllers schema as the sensor for that zone.

So if the controller is allocated as a sensor to a zone via schema.json (and it is the only sensor in the zone), would it be appropriate to assume that the zone's temperature / setpoint is the controller temperature / setpoint?

Ultimately I guess this is more a nice-to-have really, since most people would only be interested in the zones themselves.

I only asked the question originally as I initially thought "oh cool, you also get all the device individual data too" but then noticed that some facets were missing for the hall room thermostat device (which is the controller).

[zxdavb]

Note: The controller, as a device, has no concept of setpoint, because it is not an actuator (i.e. it is not a TRV, a relay, etc.).

In fact, the controller is not really a device at all is most respects, except it does have a temperature sensor, just like: TRVs, the various Thermostats, the DHW sensor, etc.

I've used the schema.json to allocate the controller as the sensor for zone 0

To be clear - without going into details - I don't think it really offers any value to do so.

From your comment here I think you're therefore saying that it's not possible to show the temperature / setpoint of the controller device itself as it's not physically sending that over RF?

Correct (although see my note above - the controller does not have a setpoint), but you can get the temperature / setpoint of the corresponding zone: you just ask the controller (that is it's main job - to use device data & the schema to maintain the system's state).

and it is the only sensor in the zone

I want to make clear - a zone can only have 1 (or zero) sensor. It can have multiple 'child' devices - many/all of which may have temperature sensors - the controller will pay no attention to those temperatures.

So if the controller is allocated as a sensor to a zone via schema.json (and it is the only sensor in the zone), would it be appropriate to assume that the zone's temperature / setpoint is the controller temperature / setpoint?

This is incorrect. A Zone's temperature & setpoint is from the controller as a controller, not as a sensor. Let me rephrase:

If a zone has no reported temperature sensor in the schema, but the controller reports that the zone has a current temperature (and so must have a temperature sensor), is it appropriate to assume that the zone's sensor is the controller?

Answer: Yes (beware multiroom mode).

As I've said - there is little advantage is manually adding the controller (as a sensor) to the schema.json file - doing so does not provide any additional (reliable) data/functionality & can only break things (that's why I've taken the code out that was able to do this).

I only asked the question originally as I initially thought "oh cool, you also get all the device individual data too" but then noticed that some facets were missing for the hall room thermostat device (which is the controller).

This works both ways - there is some missing device data (e.g. the controller's temperature sensor) and some missing zone data (e.g. each zone's individual heat demand- evohome_rf derives this in a similar way to the controller (i.e. using a schema & device data).

[martynwendon]

Interesting stuff and clearly a lot to try and wrap your head around!

there is little advantage is manually adding the controller (as a sensor) to the schema.json file - doing so does not provide any additional (reliable) data/functionality & can only break things

Apologies for perhaps appearing dumb, but does that mean we shouldn't be adding this in the zones section of the schema now?

From previous discussions (e.g. https://github.com/zxdavb/evohome_rf/issues/15 ) I was under the impression that you had to add the controller as a sensor to schema.json in order to identify it as the sensor for the zone?

For reference my current schema is below.

schema.txt

It sounds like leaving the controller to just be a controller is a better setup and gives a fuller data set to work with.

I do have a spare DTS92 room thermostat so I might just use that for the hall zone instead. I think I only used the controller originally to save buying another room thermostat, but somehow ended up with an additional one anyway.

[zxdavb]

From previous discussions (e.g. zxdavb/evohome_rf#15 ) I was under the impression that

I am sorry I cannot exactly recall why I gave that advice then.

From previous discussions I was [given] the impression that you [have] to add the controller as a sensor to schema.json in order to identify it as the sensor for the zone?

No. Just leave it. evohome_rf can cope.

I do have a spare DTS92 room thermostat so I might just use that for the hall zone instead

This is a better option for several reasons - for one: you will get a more accurate temperature.

[martynwendon]

No problem, I know this is a constantly evolving at the moment :-)

No. Just leave it. evohome_rf can cope.

Cool, so sounds like it's best to just use schema.json to set the initial controller id and allowlist with nothing else? Presume that means it then does discovery at each restart to figure everything out?

On the flip side, having a "correct" schema.json doesn't hurt (but don't assign the controller to a zone if you're using it as a sensor)? And in that case no discovery is performed at start?

This is a better option for several reasons - for one: you will get a more accurate temperature.

I'll switch to the DTS92 I think, need to read the manuals again as I don't recall if you have to delete the zone to remove the controller as its sensor, or can just bind in a new sensor and it will sort itself out.

[zxdavb]

Cool, so sounds like it's best to just use schema.json to set the initial controller id and allowlist with nothing else?

Yes, except is is now allow_list.

Presume that means it then does discovery at each restart to figure everything out?

yes, but this is expensive & I am adding a new save state/restore state feature that will leverage a schema file.

On the flip side, having a "correct" schema.json doesn't hurt (but don't assign the controller to a zone if you're using it as a sensor)? And in that case no discovery is performed at start?

That is the plan - I am not sure if it is broken or working presently - it is a very complex piece of code.

can just bind in a new sensor and it will sort itself out

The above is the way forward.

[martynwendon]

@smar000 just thought I'd update here, I've been running this for around a month now in parallel to my old Domitcz integration and it's proven pretty reliable, no crashes or weird behaviour.

I had some work to do to map from your topic structure to my existing setup that I've been using from Domoticz, but once that was done it's been a much more informative system due to having data from so many parts of the evohome setup (compared to that really old Domoticz integration I've been using).

There's a couple of questions, not sure if they're best aimed at yourself or @zxdavb

1) Domoticz had an "RSSI" indication for some of the devices but I've not seen that here, is it available? It's no big deal .... not like I really used it for anything so only really commenting as it's conspicuous by its absence.

2) What's the difference between "heat demand" and "relay demand"? I see relay demand from the BDR91, controller and room thermostats and heat demand from from the controller and TRVs.

3) For the "window open" feature I like that it's exposed as a sensor, but Domoticz seemed to use that to adjust the "mode" of the zone to "Open Window" which was also useful (and IIRC matches what you see on the display of the controller itself). Should that happen here too? I've not specifically tested for it yet but my current logic looks for that to do TTS announcements ("Hall Heating Zone is Open Window") and I've not noticed any for a while, so again it's conspicuous by its absence.

I guess final question is more for @smar000 as to if / when gw_evohome_rf might get updated to use the later evohome_rf library, not that there's any rush really since it's working as-is so far!

Thanks to both of you for your work on these projects!

[smar000]

Thanks for the update @martynwendon and good to hear that is been reliable for you.

In answer to your queries:

1. RSSI is present in the framework library, but just not being displayed by my script. I can add it in a future update.

2. I am probably not the best person to tell you the technical differences, but from my simplistic understanding, the two are probably similar - just for different device types, and in the case of the relay demand, an amalgamation of all heat demands. For example, you may have multiple TRVs demanding heat. Each will report just it's own demand as a percentage of its own zone's setpoint etc. The relay demand then shows a combined net demand on the relay itself, i.e. what percentage of time it needs to be open to fulfill the heat demand from all the TRVs. I could be totally wrong, but that's how I've always interpreted it!

3. From my understanding, the changing of the mode to window open is something done at the controller level and not by David's framework or my wrapper code. Are you sure the mode change is being done by Domoticz itself? Anyway, I had a quick look at my logs and it does seem to be correctly picking up window state messages. You should be able to trigger a rule of these notifications to do your TTS announcements:

2021-04-16 18:50:44,720 [339] || TRV:228044 | CTL:139901 |  I | window_state     | 000000   || {'parent_idx': '00', 'window_open': False}
2021-04-16 19:00:54,366 [339] || TRV:227952 | CTL:139901 |  I | window_state     | 010000   || {'parent_idx': '01', 'window_open': False}

In answer to your final query, I have been a bit tied up with life/work recently so had put everything on hold (particularly as David mentioned that there would be a number of breaking changes introduced in the later builds). I've updated locally to the current release of the framework and will (time permitting) try to get it updated in the coming weeks.

[martynwendon]

Thanks for the reply @smar000 and no problem with timescales on future updates, everything is functional for me as-is in any case.

With the heat demand vs relay demand that sounds similar to my understanding i.e. the TRVs have a heat demand and then that's reflected by a relay demand seen on the controller and bdr91. The oddity was that I also see a relay demand on the room thermostats, which made me then question my own understanding of how things worked :-)

I do see the window_state so that's fine (I've mapped them to a "temperature drop" sensor in my own system which is nice). It was mostly that my current TTS announcements had stopped (or at least, I haven't heared any for a while, which was odd as the kids are always leaving their windows open so I'd hear them several times a day usually). Those are currently based on the mode of the zone changing from auto / whatever to "open window". I'll dig a bit deeper into that to see what's going on, I just wondered if that "open window" mode was a feature of the system or something that was faked in Domoticz.

[zxdavb]

Domoticz had an "RSSI" indication for some of the devices

This is layer one (packets), but is available. We mainly deal in layer two (messages) and three (payloads). This is in addition to the rf_check packet, which is still available.

What's the difference between "heat demand" and "relay demand"?

One is a request (heat), the other is an action (relay).

Each actuator can send the controller a heat demand (a call for heat) - the controller constructs the heat demand for each zone from that information, moderated by any optimizations, etc. (this is not trivial). The system heat demand is then calculated from that, and becomes a relay demand instruction to the boiler controller (FC, +/- any other relays F9/FA). The relay will announce the demand it is doing (which hopefully matches what was asked of it).

[ TIP: the primary difference between an electric zone, and a zone valve zone is that an EZ's heat demand is ignored when constructing the system's overall heat demand. That is, an EZ cannot cause a call for heat.]

what percentage of time it needs to...

Of course, for OpenTherm, the system heat demand is not a % of the TPI cycle, but is more complex.

The controller object has a heat_demands object, and a relay_demands object - have a look. Be clear - there is a difference between what the zones request, what the controller instructs, and what the relays (FC, and F9/FA) do.

The oddity was that I also see a relay demand on the room thermostats

An anachronism.

For the "window open" feature I like that it's exposed as a sensor, but Domoticz seemed to use that to adjust the "mode" of the zone to "Open Window"

Open window is a state, not a mode. it 'overrules' any zone override, which itself takes precedence over any schedule/mode pair (e.g. scheduled setpoint,/Away mode).

The controller does not reveal if a zone is ever in OpenWindow state, or not - it is inferred by evohome_rf via eavesdropping packets sent from TRVs... If evhome_rf picks up one of these packets, and the controller does not, then is evohome_rf wrong (c.f. controller UI) or right (c.f. TRV behaviour)?

[martynwendon]

The relay will announce the demand it is doing (which hopefully matches what was asked of it).

Got it, that was what I understood previously - the TRV's have the heat demand and the controller constructs an overall heat demand for each zone and then from that a relay demand, which it sends to the BDR91 in my case. BDR91 has a relay demand which should then match what was requested.

An anachronism.

So do the relay demands from the room thermostats not actually represent anything then? The data does seem to change, for example:

"/evohome/gateway/_zone_independent/thm_22:046163/relay_demand/relay_demand": [
                "0.0",
                "0.29",
                "0.27",
                "0.24",
                "0.22",
                "0.19",
                "0.16",
                "0.13",
                "0.11",
                "0.08",
                "0.05",
                "0.03"
        ],

Or is that perhaps because these room thermostats can also function as a standalone entity too, i.e,. paired directly to a BDR91? So while this data is present, it's not actually used within the evohome system at all?

The controller does not reveal if a zone is ever in OpenWindow state, or not - it is inferred by evohome_rf via eavesdropping

So does that mean that the zone mode IS changed to OpenWindow (by evohome_rf) when window_open state from the TRV is true? Or is that up to the end user to implement that?

I'm good either way, it's trivial to do it myself but if it's already done by evohome_rf then that saves me from doing it :-)

[zxdavb]

the relay demands from the room thermostats not actually represent anything then... Or is that perhaps because these room thermostats can also function as a standalone entity too, i.e,. paired directly to a BDR91? So while this data is present, it's not actually used within the evohome system at all?

The DTS92(E) has a setpoint, and it knows the temperature: it calculates a heat demand between the two and broadcasts a relay demand - AFAICT, evohome simply ignores those broadcasts.

zone mode IS changed to OpenWindow (by evohome_rf) when window_open state from the TRV is true?

The zone's mode and state is reported by evohome_rf - it can change a zone's mode (e.g. to TemporaryOverride), but only reports the OpenWindow state. The controller will not provide this info, so evohome_rf uses the same source of data the controller does - packets broadcast by TRVs, hopefully:

• evohome_rf sees the same packets as the controller
• the algorithm evohome_rf uses on this data is the same as the controller's

[martynwendon]

@smar000 I have everything mapped across nicely now so incoming data is all working fine.

Outgoing I'm not having much luck with though, for example changing the system mode, zone setpoints, with errors such as "get_controller_mode_payload is not defined".

Peeking at the code none of the methods called from build_send_string() seem to exist, so perhaps sending isn't implemented yet? Appreciate this is all very much WIP so no problem if that's all still on the todo list!

[smar000]

Hi @martynwendon

Yes, the send side was put on hold as David was going through some breaking changes on the framework (many related to the send command side of things) at the time.

It is very much on the to-do, and I'm hoping to get back to it in the coming week(s).

[martynwendon]

No problem, I still have the old Domoticz instance running in parallel so I can continue using that for "sending" for the time being.

[smar000]

No problem, I still have the old Domoticz instance running in parallel so I can continue using that for "sending" for the time being.

Great! I'll ping you here once I have the send side uploaded.

[zxdavb]

From my point of view, evohome-rf is now stable enough for you to proceed when you're ready.

[smar000]

From my point of view, evohome-rf is now stable enough for you to proceed when you're ready.

That's great, thanks for letting me know.

[smar000]

Hi @martynwendon

Just to let you know that I've just uploaded an updated build which should support sending commands. It is still a "work in progress", but for the most part seems to be working (support for failed send commands is not yet included). Please see the README for changes to the send command message structure etc.

Thanks.

[martynwendon]

@smar000 that's great, I'll check it out at the weekend.

So far the current build has been rock solid and hasn't missed a beat!

[smar000]

Hi @martynwendon

As the migration has been moved to master, I'll close this issue for now. If you have any further issues, please do not hesitate to open another issue.