Exposure as Transport

[smar000]

Hi @zxdavb

Not an issue, but I'm the author of evogateway/listener - you may recall we had a brief chat a few months back in the automatedhome forum. I was wondering if you've had any further thoughts on exposing your library as a transport. No urgency!

Many thanks.

[zxdavb]

Hi @smar000

It is fully exposed as a transport, using the standard asyncio Protocol/Transport model.

Have a look at client.py, or - more simply:

from evohome_rf import Gateway

def process_message(msg) -> None:
    # e.g. msg.payload is the the message dict, msg.src.id is the source device address
    dtm = f"{msg.dtm:%H:%M:%S.%f}"[:-3]
    print(f"{dtm} {msg}")

async main(serial_port, **kwargs):
    gwy = Gateway(serial_port, **kwargs)
    protocol, _ = gwy.create_client(process_message)

    await asyncio.create_task(gwy.start())

if __name__ == "__main__":
    asyncio.run(main("/dev/ttyUSB0"))

The idea is - as I'm sure you'll now - is to add your evohome_rf --> RESTful gateway in process_message().

Receiving messages (an inbound msg) is pretty good, but sending commands (an outbound msg) isn't fully fleshed out yet - you can use gwy.send_data("RQ ...") for now.

There are a shedload of config parameters available (see lib_kwargs in client.py and also schema.py) - reduce_processing and the allow/block lists are a good start, I guess.

Workflow is a bit broken, presently: I'll tidy up the master branch today/tomorrow & move dev to another branch.

Any questions, just ask!

[smar000]

Thanks for the quick response back, and that's great. I haven't actually had a chance to look at your latest code but will hopefully do so over the weekend.

Thanks again and I'm sure I will have questions... :)

[zxdavb]

No worries: I'd be very happy to see someone else using it!

[smar000]

You have clearly done a tremendous amount of work - really impressive!!

One small issue - I am getting the following error/stack trace:

Traceback (most recent call last):
  File "./evohome2mqtt.py", line 65, in <module>
    asyncio.run(main("/dev/evoftdi"))
  File "/usr/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.7/asyncio/base_events.py", line 579, in run_until_complete
    return future.result()
  File "./evohome2mqtt.py", line 59, in main
    await asyncio.create_task(gwy.start())
  File "/opt/scripts/evohome/evohome_rf/evohome_rf/__init__.py", line 157, in start
    serial_port=self.serial_port,
  File "/opt/scripts/evohome/evohome_rf/evohome_rf/transport.py", line 720, in create_pkt_stack
    ser_instance.set_low_latency_mode(True)  # only for FTDI?
AttributeError: 'Serial' object has no attribute 'set_low_latency_mode'

Commenting out the respective lines in transport.py allows me to continue, without any noticeable issues. I am using a couple of nanoCUL devices. pyserial-asyncio is version 0.5.

[zxdavb]

Thanks - I will fix this - it seems your nanoCUL is not FTDI?

I had a look at your code - this library will do a lot of the heavy lifting for you - designed exactly for your use-case. Good luck!

Hmmm. Should be:

    if os.name == "posix":
        try:
            ser_instance.set_low_latency_mode(True)  # only for FTDI?
        except AttributeError:  # TODO: also: ValueError?
            pass

Anyway, removing that code block will do no harm.

[smar000]

Yes absolutely - I should be able shrink my code down signficantly.

WRT the latency issue, I tried with 2 nanoCULs. One with an FTDI and one with CH340. Both gave the same issue, so I'm not entirely sure that it is FTDI related.

Anyway, its running without the lines, and so not a big issue!

[smar000]

Hi @zxdavb

I have got a skeleton version of my system working with your library - looking great so far. Would you mind giving me a full sample scheme showing controller, dhw, ufh controller, trvs, zones etc (I had a look at schema.py but it would be quicker if I had a sample to look at), but haven't been able to get device names to show correctly.

Also are you open to exposing some further properties/attributes, e.g. in Message object, in the __repr__ function, you have some useful bits that could be re-used, e.g, src, dst, code_name, payload from the code below (appropropriately renamed so as not to clash with the main class properties of the same name):

        if self.src.id == self.devs[0].id:
            src = display_name(self.src)
            dst = display_name(self.dst) if self.dst is not self.src else ""
        else:
            src = ""
            dst = display_name(self.src)

        code_name = CODE_NAMES.get(self.code, f"unknown_{self.code}")
        payload = self.raw_payload if self.len < 4 else f"{self.raw_payload[:5]}..."[:9]

Thanks!

[martynwendon]

I have got a skeleton version of my system working with your library - looking great so far

@smar000 apologies for barging in on the thread, but, I was literally just looking at seeing if evohome_rf could be used for a project to integrate directly with MQTT so your question is great timing.

Currently using an HGI80 on a really old and heavily customised Domoticz install just for evohome. I've been considering migrating to Home Assistant but my whole system is modular built using node-red & MQTT, so while moving to HA would get me an up-to-date evohome integration I'm trying to move away from using a myriad of different smart home software just to provide specific integrations. I'd much rather have hardware <-> MQTT directly!

Is your evohome2mqtt project public anywhere? Happy to help with testing in return for access, I've got a spare HGI80 that I used for development in the past so would run with that so as not to impact my live system.

[smar000]

Hi @martynwendon I have an existing evohome -> MQTT system in my repo (evolistener/gateway), where I rewrote a lot of the domoticz packet decoding/encoding to python. That system is working reasonably well.

Having looked at @zxdavb work here though, he has a built a much more robust - and comprehensive - packet decoding/encoding library, and so it makes sense for me to migrate over to his library.

My current 'skeleton' system is no where near ready for public use, and will be many weeks/month or so away before I upload it publicly.

[smar000]

@zxdavb I came across an interesting error a short while ago:

16:17:12.784 000 RQ --- 01:191706 13:102710 --:------ 0008 001 00 < Validation error: Invalid verb/code for 01:191706: RQ/0008
16:17:12.786 044 RP --- 13:102710 01:191706 --:------ 0008 002 00C8 < CorruptStateError
16:17:12.786 handle_exception(): Caught: Exception in callback SerialTransport._read_ready()
16:17:12.786 Unhandled error in exception handler
context: {'message': 'Exception in callback SerialTransport._read_ready()', 'exception': CorruptStateError(CorruptStateError(...), '13:102710 (BDR) has changed controller: 01:139901 to 01:191706'), 'handle': <Handle SerialTransport._read_ready()>}
Traceback (most recent call last):
  File "/usr/lib/python3.7/asyncio/base_events.py", line 1639, in call_exception_handler
    self._exception_handler(self, context)
  File "/usr/local/lib/python3.7/dist-packages/evohome_rf-0.5.5-py3.7.egg/evohome_rf/__init__.py", line 124, in handle_exception
    raise exc
  File "/usr/lib/python3.7/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/local/lib/python3.7/dist-packages/serial_asyncio/__init__.py", line 106, in _read_ready
    self._protocol.data_received(data)
  File "/usr/local/lib/python3.7/dist-packages/evohome_rf-0.5.5-py3.7.egg/evohome_rf/transport.py", line 408, in data_received
    self._data_received(*create_pkt(line))
  File "/usr/local/lib/python3.7/dist-packages/evohome_rf-0.5.5-py3.7.egg/evohome_rf/transport.py", line 612, in _data_received
    self._callback(pkt)  # only wanted PKTs up to the MSG transport's handler
  File "/usr/local/lib/python3.7/dist-packages/evohome_rf-0.5.5-py3.7.egg/evohome_rf/protocol.py", line 139, in _pkt_receiver
    [p.data_received(msg) for p in self._protocols]
  File "/usr/local/lib/python3.7/dist-packages/evohome_rf-0.5.5-py3.7.egg/evohome_rf/protocol.py", line 139, in <listcomp>
    [p.data_received(msg) for p in self._protocols]
  File "/usr/local/lib/python3.7/dist-packages/evohome_rf-0.5.5-py3.7.egg/evohome_rf/protocol.py", line 368, in data_received
    self._callback(msg)
  File "/usr/local/lib/python3.7/dist-packages/evohome_rf-0.5.5-py3.7.egg/evohome_rf/message.py", line 550, in process_msg
    create_devices(msg)  # from pkt header & from msg payload (e.g. 000C)
  File "/usr/local/lib/python3.7/dist-packages/evohome_rf-0.5.5-py3.7.egg/evohome_rf/message.py", line 378, in create_devices
    this._gwy._get_device(this.src, ctl_addr=this.dst)
  File "/usr/local/lib/python3.7/dist-packages/evohome_rf-0.5.5-py3.7.egg/evohome_rf/__init__.py", line 234, in _get_device
    device._set_ctl(ctl)
  File "/usr/local/lib/python3.7/dist-packages/evohome_rf-0.5.5-py3.7.egg/evohome_rf/devices.py", line 201, in _set_ctl
    f"{self} has changed controller: {self._ctl.id} to {ctl.id}"
evohome_rf.exceptions.CorruptStateError: The system state is inconsistent: 13:102710 (BDR) has changed controller: 01:139901 to 01:191706 (try restarting the client library)

I certainly haven't changed controllers, and don't recognise 01:191706 as one of my devices... I don't believe any of my neighbours have an evohome system either.

Any ideas?

Thanks.

[martynwendon]

Hi @martynwendon I have an existing evohome -> MQTT system in my repo (evolistener/gateway), where I rewrote a lot of the domoticz packet decoding/encoding to python. That system is working reasonably well.

I think I did try this a few years ago but didn't have much success with the HGI80. I don't recall what the problem was, but I didn't have a spare HGI80 back then so didn't spend too much time on it. I stayed with Domoticz as that was already working.

I'll take another look though since I have the spare HGI80 now so can afford to play around a bit more.

Having looked at @zxdavb work here though, he has a built a much more robust - and comprehensive - packet decoding/encoding library, and so it makes sense for me to migrate over to his library.

Agreed, the progress has been amazing and I've been tracking the thread on the Home Assistant forum for quite some time. I had a bit of spare time so thought I'd dig into possibly writing my own MQTT direct gateway using the evohome_rf library, hence I spotted your question.

My current 'skeleton' system is no where near ready for public use, and will be many weeks/month or so away before I upload it publicly.

No problem at all, I'll check back at a later date. In the meantime if you do want anybody to test please let me know.

[zxdavb]

Would you mind giving me a full sample scheme showing controller, dhw, ufh controller, trvs, zones etc

This is a cut and past job - will give the idea - note the mapping between UFH circuit and a evohome zone

Schema[01:145038 (evohome)] = {
    "controller": "01:145038",
    "system": {
        "heating_control": "13:237335",
        "orphans": [
            "04:225661",
            "04:225663"
        ]
    },
    "ufh_controllers": {
        "02:000921": {
            "ufh_circuits": {
                "00": {
                    "zone_idx": "03"
                },
                "01": {
                    "zone_idx": "06"
                },
                "02": {
                    "zone_idx": "04"
                }
            }
        },
    },
    "stored_hotwater": {
        "hotwater_sensor": "07:017494",
        "hotwater_valve": "13:109598",
        "heating_valve": null
    },
    "zones": {
        "00": {
            "heating_type": "zone_valve",
            "sensor": null,
            "devices": [
                "13:163420",
                "13:064044"
            ]
        },
        "01": {
            "heating_type": "underfloor_heating",
            "sensor": "34:190267",
            "devices": [
                "34:190267"
            ]
        },
...
        "0B": {
            "heating_type": "radiator_valve",
            "sensor": "22:098449",
            "devices": [
                "04:098449"
            ]
        }

[zxdavb]

Also are you open to exposing some further properties/attributes

Everything you might need should be exposed, or example:

msg.verb, msg.src.id, msg.dst.id, msg.code, msg.payload = (
    "RQ", "01:123456", "13:234567", "3B00", {"...
)

Is there anything specific you think you might need, but you can't find?

[zxdavb]

Invalid verb/code for 01:191706: RQ/0008

This is simply saying that an 01: device is not expected to send an RQ/0008 (which is true - they don't normally do this) - this suggests a corrupt pkt.

evohome_rf.exceptions.CorruptStateError: The system state is inconsistent: 13:102710 (BDR) has changed controller: 01:139901 to 01:191706 (try restarting the client library)

This confirms the diagnosis - the source address is corrupt (t was likely the HGI, which evohome_rf is getting to send RQ/0008 pkts).

This unfortunately common, and I have to work out a way of handling this. The controller doesn't have this issue, because it knows teh 'truth', whilst evohome_rf relies upon eavesdropping.

[zxdavb]

I am not sure if you need to have evohome_rf maintain state? Possibly you can do the equivalent of this:

python client.py listen -rr /dev/ttyUSB0 ...

... and just consume (i.e. forward via MQTT) the messages?

Have you tried to do anything like:

python client.py monitor /dev/ttyUSB0 -o packet.log -x "RQ 01:123456 1F09 00"

...and I'm always keen on receiving packet logs to test against!

[smar000]

Many thanks for the above - very helpful, and noted on the likely cause of the error message.

Have you tried to do anything like:

I haven't yet, but I should have the packet logs saved. Let's see how I get on, and if the problem persists, I will get back to you.

Everything you might need should be exposed

Thanks and yes, for the most part I found they are. One specific one that I didn't fine is the code_name in a Message. I had to lookup the name by importing CODE_NAMES from .message and checking against this.

Not a big deal TBH, but I thought it might be helpful to have this within the Message class itself.

Thanks again.

[zxdavb]

You can use a packet log as a source, rather than a serial port - the will be useful fro test/dev:

cat packet.log | python client.py parse

I think code_name will be a useful addition!

Re: Opentherm - I am driven/limited by other projects, from which I am reluctant to deviate far... I see that JSON as distinct from my schema. E.g. https://github.com/mvn23/pyotgw

[smar000]

Re: Opentherm - I am driven/limited by other projects, from which I am reluctant to deviate far... I see that JSON as distinct from my schema. E.g. https://github.com/mvn23/pyotgw

Thanks and that is fine. After thinking about it, and seeing the different results under opentherm_msg, I think the way you have done it is logical. I have coded to take this into account.

With regards to sending commands, is there any callback that confirms the status of the sending, e.g. command sent/success/fail notifications? I see various packet messages showing retry/expired etc, but haven't (yet) looked to see how it all works in your code.

EDIT: Also is there any way to 'label' or tag devices? For example, if say there are multiple TRVs in a room, can we identify specific TRVs by name?

[smar000]

Another question - in your schema, what is the correct way to define underfloor_heating zone? In your example above, you gave:

 "01": {
            "heating_type": "underfloor_heating",
            "sensor": "34:190267",
            "devices": [
                "34:190267"
            ]
        },

1. If I use the temperature sensor in the main controller itself for one zone, I assume we would put the controller ID for the sensor in that zone definition?

2. If I have an HCC80R controlling UFH, how is this specified under devices for each individual zone, particularly given that tthe HCC80R has its own 'sub-zones' (as identified by ufh_idx in your message responses)? Would these sub-zones need to be specified? If so, where does this go?

Hope the above makes sense!

Thanks.

[zxdavb]

OMG, the answer to this will be worth making into a wiki entry...

1st (useful to know): the output schema can be used as an input schema, so just use:

python client.py monitor /dev/ttyUSBx -x "RQ 01:xxxxxx 1F09 00"

and hit Ctrl-C after the last RP/01:xxxxxx/0418/xx - the schema should be there for you. Have you run the CLI yet - to do so is very useful, so get involved in that.

2nd: I have not had a UFH system to test against, so expect some issues. To answer this question:

If I have an HCC80R controlling UFH, how is this specified

Please start by providing me the schema, so we can discuss it without confusion.

3rd (the biggie): There are three means used by evohome_rf to construct the schema (in order of development):

• from eavesdropping (by capturing packets between the controller and other devices)
• from the input schema provided as a dict
• from discovery (i.e. the controller is asked a series of RQ s, and the corresponding RP are used)

There are advantages to all 3 options - and each has issues, too. One is that when an received packet - usually an eavesdropped packet) does not match the schema: CorruptStateError (another reason why your input schema should be 'right').

The schema is important, it is used to construct some meta-data (e.g. zone heat demand from the child devices of that zone), and the objects (so you can, e.g. set the mode of a zones).

4th...

If I use the temperature sensor in the main controller itself for one zone

I just want to say, just want to avoid confusion: the schema should reflect reality: (evohome_rf doesn't change the behaviour of evohome), so 'making' the controller the sensor for a zone is only appropriate if it is the sensor for that zone.

There is no simple means to learn the sensor for a zone, when that sensor is the controller - I do have some code for this, but it is only 'nearly-deterministic' and is currently disabled. That is, the only 'hard/fast' way to know which zone the controller is a sensor for (if any) is via an input schema (which we hope is 'true' to reality for the above reasons).

[zxdavb]

AttributeError: 'Serial' object has no attribute 'set_low_latency_mode'

What version of pyserial / pyserial-asyncio are you using?

[smar000]

Thanks for the above - very helpful. I did try the cli, but didn't realise that it built the full schema for me - definitely one for the wiki!

Schema output from discovery below:

Schema[01:139901 (evohome)] = {
    "controller": "01:139901",
    "system": {
        "heating_control": null,
        "orphans": []
    },
    "ufh_controllers": {
        "02:043392": {
            "ufh_circuits": {
                "00": {
                    "zone_idx": "07"
                },
                "01": {
                    "zone_idx": "0A"
                },
                "02": {
                    "zone_idx": "0B"
                }
            }
        }
    },
    "stored_hotwater": {
        "hotwater_sensor": "07:043555",
        "hotwater_valve": "13:133904",
        "heating_valve": "13:102710"
    },
    "zones": {
        "00": {
            "heating_type": "radiator_valve",
            "sensor": "34:015243",
            "devices": [
                "04:228044",
                "04:143928",
                "04:143926",
                "04:143922"
            ]
        },
        "01": {
            "heating_type": "radiator_valve",
            "sensor": "04:143924",
            "devices": [
                "04:227952"
            ]
        },
        "02": {
            "heating_type": "radiator_valve",
            "sensor": "04:228096",
            "devices": [
                "04:228042"
            ]
        },
        "03": {
            "heating_type": "radiator_valve",
            "sensor": "04:227954",
            "devices": [
                "04:227954"
            ]
        },
        "04": {
            "heating_type": "radiator_valve",
            "sensor": "04:000868",
            "devices": [
                "04:000868"
            ]
        },
        "05": {
            "heating_type": "radiator_valve",
            "sensor": "04:228048",
            "devices": [
                "04:228048"
            ]
        },
        "06": {
            "heating_type": "zone_valve",
            "sensor": "34:231315",
            "devices": [
                "13:230917"
            ]
        },
        "07": {
            "heating_type": "underfloor_heating",
            "sensor": null,
            "devices": []
        },
        "08": {
            "heating_type": "radiator_valve",
            "sensor": "04:001048",
            "devices": [
                "04:001048"
            ]
        },
        "09": {
            "heating_type": "radiator_valve",
            "sensor": null,
            "devices": [
                "04:228094"
            ]
        },
        "0A": {
            "heating_type": "underfloor_heating",
            "sensor": "34:112193",
            "devices": []
        },
        "0B": {
            "heating_type": "underfloor_heating",
            "sensor": "34:214769",
            "devices": []
        }
    }
}

I have an HCC80R installed, and so can give you any data that you may need from this.

Also in answer to your 4th point, yes, the scenario is "actual", in that my Dining Room zone (with ID "07" in the schema) is UFH, whose actuators are controlled by HCC80R, and whose temperature sensing is done by the main controller, which is situated in the dining room. I noticed that in the auto-discovered schema above, it is not showing any sensor for this zone "07"... I don't know if that is because it is confused with the controller being there, or may be I did not run the client long enough?

Given the importance of getting the schema correct, it might be helpful for end users if there was a pre-defined command for the cli that just went and got the schema and saved it to a file (or stdout) once the schema was complete. WDYT?

[smar000]

AttributeError: 'Serial' object has no attribute 'set_low_latency_mode'

What version of pyserial / pyserial-asyncio are you using?

pyserial is 3.4, and pyserial-asyncio is version 0.5

[smar000]

Just a follow on WRT the schema file. I added the above schema and tried to run the client with the schema as input. It is giving me the following error for the ufh_controller section:

client.py: Starting evohome_rf...
Traceback (most recent call last):
  File "client.py", line 370, in <module>
    cli()
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "client.py", line 199, in monitor
    asyncio.run(main(lib_kwargs, command=MONITOR, **cli_kwargs))
  File "/usr/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.7/asyncio/base_events.py", line 579, in run_until_complete
    return future.result()
  File "client.py", line 320, in main
    gwy = Gateway(lib_kwargs[CONFIG].pop(SERIAL_PORT, None), **lib_kwargs)
  File "/opt/scripts/evohome/evohome_rf/evohome_rf/__init__.py", line 103, in __init__
    self._schema, self.known_devices = load_schema(self, **kwargs)
  File "/opt/scripts/evohome/evohome_rf/evohome_rf/schema.py", line 216, in load_schema
    schema = SYSTEM_SCHEMA(kwargs.get("schema", {})) if kwargs.get("schema") else {}
  File "/usr/local/lib/python3.7/dist-packages/voluptuous/schema_builder.py", line 272, in __call__
    return self._compiled([], data)
  File "/usr/local/lib/python3.7/dist-packages/voluptuous/schema_builder.py", line 594, in validate_dict
    return base_validate(path, iteritems(data), out)
  File "/usr/local/lib/python3.7/dist-packages/voluptuous/schema_builder.py", line 432, in validate_mapping
    raise er.MultipleInvalid(errors)
voluptuous.error.MultipleInvalid: not a valid value for dictionary value @ data['ufh_controllers']

This is the ufh_controller section in the auto-discovered schema:

"ufh_controllers": {
        "02:043392": {
            "ufh_circuits": {
                "00": {
                    "zone_idx": "07"
                },
                "01": {
                    "zone_idx": "0A"
                },
                "02": {
                    "zone_idx": "0B"
                }
            }
        }
    },

Any suggestions as to what the issue may be?

Thanks.

[zxdavb]

   "07": {
       "heating_type": "underfloor_heating",
       "sensor": null,
       "devices": []
   },

So simply change the above to have:

        "07": {
            "heating_type": "underfloor_heating",
            "sensor": "01:139901",
            "devices": []
        },

But! Don't be surprised it it doesn't work - I think I made some breaking changes - I will check it now.

Just checked: this seems to work OK (a minimal schema, the rest done by discovery):

    "schema": {
        "controller": "01:145038",
        "zones": {
            "00": {
                "sensor": "01:145038",
            }
        }
    }

Later, I'll test a full schema & let you know.

Of course, you can test it too.

[zxdavb]

I noticed that in the auto-discovered schema above, it is not showing any sensor for this zone "07"... I don't know if that is because it is confused with the controller being there, or may be I did not run the client long enough?

As I said, above: there is no deterministic way to work out what zone the controller is a sensor for (if any) - you have to analyse the meta-data, and even then it's not fully guaranteed to be 'truth'.

The best scenario is that you have N+1 zones (presumably all with sensors), and only N sensors (other than the controller).

[zxdavb]

Given the importance of getting the schema correct, it might be helpful for end users if there was a pre-defined command for the cli that just went and got the schema and saved it to a file (or stdout) once the schema was complete. WDYT?

Reasonable idea - will add it as a feature, but until then:

python client.py monitor /dev/ttyUSB0 > msg.out

TBH, you don't really need a schema, except for:

• to ID your controller (vs that of your neighbours)
• to ID the zone that the controller is a sensor for

Everything else can be discovered for evohome (that doesn't apply for other RAMSES-based systems, like Sundial, etc.).

In fact, it is probably better to keep it minimal, rather than risk it being wrong:

    "schema": {
        "controller": "01:123456",
        "zones": {
            "07": {
                "sensor": "01:123456",
            }
        }
    }

Note: evohome_rf will track all controllers (i.e. systems) it sees, but by default the first controller seen is the controller, (unless the allow/block lists come into play) - The schema allows you to ensure that this is your controller.

[zxdavb]

I need a packet log from you

python client.py monitor /dev/ttyUSB0 -o packet.log

Just a follow on WRT the schema file. I added the above schema

Any suggestions as to what the issue may be?

Let me test it (as I said, I haven't done a lot of UFH testing).

Use this for now:

    "schema": {
        "controller": "01:139901",
        "zones": {
            "07": {
                "sensor": "01:139901"
            }
        }
    }

[smar000]

Later, I'll test a full schema & let you know.

Of course, you can test it too.

Thanks and yes, I will be testing too. Packet log with above schema attached: packet.log

[smar000]

Use this for now:

    "schema": {
        "controller": "01:139901",
        "zones": {
            "07": {
                "sensor": "01:139901"
            }
        }

Even starting with just the above, autodiscovery seems to overwrite the zone 7 sensor value to null.

[zxdavb]

Yes sector makes sense in retrospect - it's an easy fix.

what's happening is that evohome-rf is asking the controller for the sensor of that zone and it's saying that there is no sensor for that zone.

On Wed, 10 Feb 2021, 15:09 smar, notifications@github.com wrote:

Use this for now:

"schema": {
    "controller": "01:139901",
    "zones": {
        "07": {
            "sensor": "01:139901"
        }
    }

Even starting with just the above, autodiscovery seems to overwrite the zone 7 sensor value to null.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/zxdavb/evohome_rf/issues/15#issuecomment-776772832, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABTYKCG6STW2EXFSFMWB4LDS6KOUFANCNFSM4XEYN6VA .

[smar000]

it's saying that there is no sensor for that zone.

Thanks and yes, I had guessed as much.

BTW, I would recommend your earlier comment https://github.com/zxdavb/evohome_rf/issues/15#issuecomment-776716794 to be also added to the wiki!

[smar000]

Would it be possible for you to remove the basicConfig configuration in packet.py:

logging.basicConfig(level=DEFAULT_LEVEL, format=DEFAULT_FMT, datefmt=DEFAULT_DATEFMT)

This seems to take over the logger in other scripts, .e.g if I create a new logger in the client.py with separate handlers for writing to file and showing on console with writing to file set at logging.DEBUG, and to console on logging.ERROR, it always writes to console, regardless of the log level in the console handler. Commenting out the above line in packets.py resolves it for me.

Thanks.

[zxdavb]

... makes sense in retrospect - it's an easy fix.

Actually, not that straight-forward at all! I am also a little troubled by having to commit to a schema - give me a few days to think about it...

Would it be possible for you to remove the basicConfig configuration in packet.py

I hope so - will have a look - the library should definitely not affect logging at the client level. Any change I can see you code - may be helpful.

I've got a small feature dump lined up too...

[smar000]

Actually, not that straight-forward at all! I am also a little troubled by having to commit to a schema - give me a few days to think about it...

No worries and will leave it with you. For my purposes, I don't think it is as much of an issue - more a nice to have (I think!).

I hope so - will have a look - the library should definitely not affect logging at the client level. Any change I can see you code - may be helpful.

Can you try the code below, with both including the from evohome_rf import Gateway and commenting it out. This will hopefully show what I mean.

import logging
from logging.handlers import RotatingFileHandler
from evohome_rf import Gateway

log = logging.getLogger(__name__)
log.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

# Log file handler
file_handler = RotatingFileHandler("./test.log", maxBytes=1000000, backupCount=2)
file_handler.setLevel(logging.DEBUG)
file_handler.setFormatter(formatter)
log.addHandler(file_handler)

# Log console handler
console_handler = logging.StreamHandler()
console_handler.setLevel(logging.ERROR)
console_handler.setFormatter(formatter)
log.addHandler(console_handler)

log.info("Info: This should be in log file only")
log.debug("Debug: This should be in log file only")
log.warning("Warning: This should in log file only")
log.error("Error: This should be both on console and in log file")

I've got a small feature dump lined up too...

Great, look forward to it!

The one feature that would be helpful to me is a callback that lets me know once a send command has completed - whether with success or failure/expired. Again, I can work around it by tracking messages at the client end, and matching send/responses etc, but as it looks as if you are already doing it in PacketProtocolQos, it would make life easier and more elegant!

[zxdavb]

I think I haven't yet answered these questions...

With regards to sending commands, is there any callback that confirms the status of the sending, e.g. command sent/success/fail notifications? I see various packet messages showing retry/expired etc, but haven't (yet) looked to see how it all works in your code.

This is not fully complete. The RAMSES protocol is based upon casting - I have been trying to - essentially - turn UDP into TCP... In short: there is an architecture; client -> Msgs -> Pkts -> serial port or (read/only) file. There is QoS - callbacks and re-transmits - built into the Pkt layer..

In addition: there is currently Pkt callbacks utilized for schedules & fault logs at the Client layer - the rest is TBD. That is you ask for a zone's schedule & it sends the required sequence of packets and you receive the schedule.

If you wish you can just hook into them at the Pkt layer. This says it all, I think:

        def rq_callback(msg) -> None:
            ....

        rq_callback = {"func": rq_callback, "timeout": td(seconds=10)}
        self._gwy.send_data(
            Command("RQ", self._ctl.id, "0418", f"{log_idx:06X}", callback=rq_callback)
        )

Question: what would you prefer: callbacks, or an async wait? I think I could do either, but I was thinking of the latter.

I have added a bunch of constructors to the Command class to start this process: The Command class will shortly have a bunch of constructors, e.g.:

    @classmethod
    def zone_setpoint(cls, ctl_id, zone_idx, setpoint: float, **kwargs):
        payload = f"{zone_idx:02X}" if isinstance(zone_idx, int) else zone_idx
        payload += temp_to_hex(setpoint)
        return cls(" W", ctl_id, "2309", payload, **kwargs)

So, you would then have code like this:

cmd = Command.zone_setpoint("01:123456", "00", 12.0)
try:
    gwy.send_cmd(cmd)
except evoFailed:
    raise

This is just the construction of a command packet - you will still be able to:

await gwy._evo.zone_by_id["00"].set_setpoint(12.0)

... but it will just be calling the above construcor.

Life is - oddly - more complicated for getters, like this:

result = await gwy._evo.zone_by_id["00"].get_setpoint()

EDIT: Also is there any way to 'label' or tag devices? For example, if say there are multiple TRVs in a room, can we identify specific TRVs by name?

Yes. You have friendly-names, & use the config file to make that happen.

[zxdavb]

The one feature that would be helpful to me is a callback that lets me know once a send command has completed

Are you interested only in getters, or setters too?

[smar000]

Are you interested only in getters, or setters too?

For the moment, getters should be enough.

[smar000]

I have been trying to - essentially - turn UDP into TCP... In short: there is an architecture; client -> Msgs -> Pkts -> serial port or (read/only) file. There is QoS - callbacks and re-transmits - built into the Pkt layer..

Thanks and this is helpful background.

you can just hook into them at the Pkt layer

Great, will look into this.

Question: what would you prefer: callbacks, or an async wait? I think I could do either, but I was thinking of the latter.

Ok with either.

So, you would then have code like this:

This would be perfect, and save me a whole bunch of coding. At the moment, I have a build_send_string() function to do that sort of pre-processing to simplify send commands to the app. Would be great if you could add in durations/until etc (e.g. for setpoints, or mode change) into these.

Yes. You have friendly-names, & use the config file to make that happen.

Yes, I found that this required the devices to be specified in the allowlist. Again, maybe one for the wiki :)

One further request (sorry!) - could you make HGI_DEV_ID user configurable.

Thanks again for all your help, and of course the great library you have developed!

[zxdavb]

Sorry - callbacks in Msg layer, QoS (e.g. re-transmits) in Pkt layer.

Would be great if you could add in durations/until etc

I'm adding in nothing which isn't already there. But it is is all there anyway.

This is command 2309 (zone_setpoint):

cmd = Command.zone_setpoint(ctl_id, zone_idx, setpoint)
msg = await gwy.send_cmd(cmd)

... and command 2349 (zone_mode):

cmd = Command.zone_mode(ctl_id, zone_idx, mode, setpoint, until)
msg = await gwy.send_cmd(cmd)

[zxdavb]

I have a build_send_string() function

You won't need it. This is an example of a Command constructor:

    @classmethod  # constructor for 000A  # TODO
    def zone_config(
        cls,
        ctl_id,
        zone_idx,
        min_temp: int = 5,
        max_temp: int = 35,
        local_override: bool = False,
        openwindow_function: bool = False,
        multiroom_mode: bool = False,
        **kwargs,
    ):
        """Constructor to set the config of a zone (c.f. parser_000a)."""

        payload = f"{zone_idx:02X}" if isinstance(zone_idx, int) else zone_idx

        assert 5 <= min_temp <= 30, min_temp
        assert 0 <= max_temp <= 35, max_temp
        assert isinstance(local_override, bool), local_override
        assert isinstance(openwindow_function, bool), openwindow_function
        assert isinstance(multiroom_mode, bool), multiroom_mode

        bitmap = 0 if local_override else 1
        bitmap |= 0 if openwindow_function else 2
        bitmap |= 0 if multiroom_mode else 16

        payload += f"{bitmap}"
        payload += temp_to_hex(min_temp)
        payload += temp_to_hex(max_temp)

        return cls(" W", ctl_id, "000A", payload, **kwargs)

[zxdavb]

One further request (sorry!) - could you make HGI_DEV_ID user configurable.

Before I say no (it is a constant), what is your use-case?

[smar000]

Thanks for the above on the command constructor etc - all very helpful, and clearly still a lot more for me to discover in your code!

One further request (sorry!) - could you make HGI_DEV_ID user configurable.

Before I say no (it is a constant), what is your use-case?

I understood from a comment in the evofw3 repo that the ID is hardcoded in the firmware to be 18:056026 (see https://github.com/ghoti57/evofw3/issues/2#issuecomment-681130505 and the subsequent comment). The ID that you are using in your const file 18:000730, hence the request.

[zxdavb]

My understanding is that the latest evofw3 FW has been changed, so that each is very likely to have a unique address.

You send a packet using the source address 18:000730, and the HGI80/evofw3 then substitutes its own address in that field. That is, although this is sent to the gateway:

RQ --- 18:000730 13:237335 --:------ 3EF0 001 00

it becomes this when it is transmitted:

RQ --- 18:xxxxxx 13:237335 --:------ 3EF0 001 00

Are you happy that it doesn't need changing?

[smar000]

My understanding is that the latest evofw3 FW has been changed, so that each is very likely to have a unique address. Are you happy that it doesn't need changing?

Oh ok. I wasn't aware of that. In that case, no need to change anything! BTW, do you have a link to that change in the ID, just out of interest?

Would this explain the unknown HGI devices that I am seeing popping up?

Do you know how often the ID changes?

[zxdavb]

Sorry for the tardy reply - I didn't see your question.

The nanoCULs ID should never change, even across restarts.

The issue is that RF is unreliable, bit-flips are common (I see about 2 a day, where the resultant packet is still valid, but the address has changed) and it is difficult for me to know how to treat packets with a new address (this issue is not limited to 18:xxxxxx addresses).

Oddly, I have just had a conversation with the evofw3 dev about this - we are hoping to mitigate this issue with evofw3.

I have means to do something about it in evohome_rf

• schemas & whitelisting of device_ids
• validation of device_type vs packet/payload
• and (coming soon) stateful inspection of packets

You only have an nanoCUL? If you have an HGI80, is it doing it with that too?

[smar000]

Thanks for the update and makes sense (in fact, I had originally assumed it was just bit flips as well). Unfortunately I don't have an HGI80 but have two nanoCULs, and so have been testing on both.

I have means to do something about it in evohome_rf

• schemas & whitelisting of device_ids

Is this the allowlist?

• and (coming soon) stateful inspection of packets

That would do it!

[zxdavb]

Is this the allowlist?

Yes.

[zxdavb]

Just pushed to master branch - you may want to use that for a while - or bleeding_edge - up to you.

[zxdavb]

You're using a different mechanism to specify an allow list, but these notes will still be useful to read: https://github.com/zxdavb/evohome_cc/wiki/Tips-for-the-configuration.yaml-file#allow-list