eTRV temperature monitoring support

[whaleygeek]

The message structure for the eTRV is as follows:

{'header': {'encryptPIP': xxxxx, 'mfrid': 4, 'productid': 3, 'sensorid': xxxx}, 'recs': [{'length': 2, 'paramid': 116, 'paramname': 'TEMPERATURE', 'paramunit': 'C', 'typeid': 144, 'value': 20.69921875, 'valuebytes': [20, 179], 'wr': False}], 'type': 'OK'}

[whaleygeek]

Also note that we need to add the productId to Devices.py

I’ve noticed that the eTRV is not recognised. A message displays stating product unknown: mfrid:0x4 prodid:0x3 sensorid:0x461 read TEMPERATURE C = 22.69921875 ADD device:0x461 Manufactuer:Energenie Product:UNKNOWN

[whaleygeek]

Ed has a pull request pending against this repo, that I hope to merge and re-test over Christmas. I understand from Ed that he has this working fine now with the eTRV's in his house.

[whaleygeek]

The eTRV's monitoring capability already works out of the box with monitor.py - I am busy with other physical layer stuff at the moment. If anyone wants to try eTRV control, work off of ed's pull request at the moment, I'll get to that part later.

[whaleygeek]

Once the device_classes branch merges back to master, this will be very easy to do (basically just write a Devices.MIHO013 device class and describe the capabilities of the device (including action commands to do things like set a setpoint, and getter commands to get readings for ambient temp etc)

[whaleygeek]

There is some eTRV stuff over here on Ed's fork, that might inform our thinking about how to model an eTRV in the new device classes.

https://github.com/whaleygeek/pyenergenie/pull/17

[whaleygeek]

Now the device_classes sprint and branch has completed, it might be really easy to just write the ENER013 device class that describes the eTRV.

[whaleygeek]

As noted in #78 this code from Energenie has a eTRV_menu program that will probably have all the necessary info about communicating with the eTRV

http://blog.energenie4u.co.uk/tech-gadgets/something-weekend-two-way-raspberry-pi-transceiver-board-etrv-guide/

[tonbut]

Hi David, thanks for all your work on this project. I've successfully used it to monitor power using MIHO006 and controlled a switch (actually controlling storage heaters). I'm now trying to get my MIHO013 radiator valves working and looking at the code I see there is some work to do. I'm happy to take a look at this. I see the above post from the energenie blog and I have the C code so i'll see what I can get working.

[tonbut]

I have the C code running but the download of ENER314-RTprograms from Energenie web site doesn't have the eTVR_menu directory only an eTVR directory. The hope executable in there does receive messages from MIHO013 but doesn't have the menu to send messages. Is there another download somewhere? Thanks

[whaleygeek]

Hi @tonbut well, there's been various chatter about it all over the place - in some issues, on some pull requests. Some of the chatter was over here I think:

https://github.com/whaleygeek/pyenergenie/pull/17

Also some work was going on over here:

https://github.com/whaleygeek/pyenergenie/issues/78

I'll try and pull everyone over onto this single issue, so we can take stock of where everyone has got to, and perhaps push this forward to a conclusion.

[pvanderlinden]

Unfortunately I hadn't had time lately, I have an experiment branch. The code is working, the code is really messy though, for all supported messages, see: https://github.com/pvanderlinden/pyenergenie/blob/hack/src/send_on_recv.py

[gpbenton]

Hi @tonbut , I have mentioned it before, but I have working code talking to my eTRVs at https://github.com/gpbenton/engMQTTClient.

[tonbut]

Thank you guys. This is great - just what I need.

[tonbut]

Hi guys, sorry I've been quiet for a while I was tied up on other stuff but I've just managed to make a little bit of progress on getting the MIHO013 eTVR working. So I wanted to check with whaleygeek and check I'm doing things right.

These are some changes I've made to the code base: 1) added a handle_message() method to MIHO013 device that transmits a battery level request message 2) added a battery level message template to top of devices.py using same structure as existing 3) changed init.py loop() to break out of loop as soon as message is received

I needed change3 because otherwise it appears the message is sent to late for eTVR to be listening.

Now when I run test code I always get a second message from eTVR immediately after first temperature reading (every 5 mins) with the battery level. It doesn't look like the actual voltage is decoded yet.

I plan to try and get the other messages working too if I'm on the right track with this. Also I'll abstract the send to that the request is queued until next receive.

[tonbut]

Actually I spoke to soon - console message I see is debug of the battery level message being sent not received :-(

[pvanderlinden]

I really should get this in decent state, got everything working, it is just in a really messy branch at the moment. Receiving the messages from eTRV is not 100% reliable though (but that is probably because it is wireless), which is a bit annoying with a message rate of 1 in 5 minutes.

@tonbut if you are interrested in my hacky solution you can look at the full diff of this: https://github.com/pvanderlinden/pyenergenie/tree/hack as I made a bugfix in the send/receive logic of the radio module. But mainly this: https://github.com/pvanderlinden/pyenergenie/commit/74ffb77aa02fe49b8d255480c5781caff98a9985 (not 100% sure yet if it is needed, but had less problems after that)

[tonbut]

@pvanderlinden yes I'm very interested. I'll compare. Yes the 5 min period makes it very painful.

I'm really keen to get a solution within the framework of the main codebase as I want to use other devices at the same time.

[tonbut]

This has got me stumped now. :-)

@pvanderlinden I've merged your changes and tried running your test programs. test2.py looks like the right one. I create a JSON message with correct sensor id and it appears to run, receive a message and then transmit. However I can see no effect with the exercise command or battery level checks.

I've also had no success (in a similar way) with the energenie code in eTVR_menu.

I've also tried an alternate ENER314-RT and an alternate MIHO013 just in case.

So I have the following questions if anyone can help: 1) what is correct value for encryptPIP for sending? I see both 0x100 and 0xF2 in different code bases. 2) should I expect to see the MIHO013 led flash when I send a identify message? 3) should I expect a battery level response immediately after sending a request or 5 minutes later?

@gpbenton I think next step will be to bite the bullet and get your setup working and see what it does differently.

Thanks in advance.

[pvanderlinden]

@tonbut Sorry should have explained how that branch works as it is a mess with different experiments.

• For the working experiment you have to have pyzmq installed (you should be able to just pip install that, although it might need to be able to compile the module)
• in one console run bridge.py (with sudo as it needs access as the rest of the pyenergenie source)
• in the other console run send_on_receive.py (both are in src), doesn't need sudo, to send the right messages, add/remove messages from to_send, and adjust the value for the message. Also make sure to update the sensorid for the messages as these are hardcoded to mine.

the radio module in eTRV_menu is not working for some reason, it just helped me get all the different messages.

1. EncryptPIP doesn't matter, the encryption is basicly useless on the energenie devices
2. I think so, the clearest responses I could get were: get battery level, and exercise (as you can here the device move)
3. Battery response should be send immediately, but the device only receives messages really short after it sends the temperature report.

[gpbenton]

Just to answer the other question: the Identify command does flash the LED for about 30s when it receives the command (which as @pvanderlinden says has to be milliseconds after receiving the temperature report)

[tonbut]

Thanks for your answers.

I've spent a bit more time today trying to move this forward but not had any success.

@gpbenton I've tried to build your repo but I'm failing on one step I think - I've built log4c but it doesn't seem to get installed so that the include in engMQTTClient.c finds it. Can you remember the trick? (I thought sudo make install would do it)

After trying all permutations of things I can think of, I thought maybe I'm missing something fundamental here. Maybe the MIHO013 needs to be registered properly and receive an ACK? Looking at the documentation it says it should blink red fast after a successful join but I can't get this to work using the setup_tool or discover_mihome.

I'm wondering if I need a energenie hub to set the device up properly?

Any more guidance welcome!

FYI this is the way I'm sending a message: I've added a handle_message to MIHO013 class and whenever I receive a message I do this:

payload = OpenThings.Message(MIHO013_IDENTIFY) payload.set(header_productid=self.product_id, header_sensorid=self.device_id) self.send_message(payload)

MIHO013_IDENTIFY = { "header": { "mfrid": MFRID_ENERGENIE, "productid": PRODUCTID_MIHO013, "encryptPIP": 30527, "sensorid": 0 # FILL IN }, "recs": [ { "wr": True, "paramid": OpenThings.PARAM_IDENTIFY, "typeid": OpenThings.Value.UINT, "length": 0, "value": 0 # FILL IN } ] }

Tracing through the radio call I see this all appears to work and I even see the red led flash briefly. But still no reaction from the device.

[gpbenton]

I've built log4c but it doesn't seem to get installed so that the include in engMQTTClient.c finds it

log4c.h should be installed in /usr/local/include. Did you do sudo make install for log4c?

I'm wondering if I need a energenie hub to set the device up properly?

I don't have an energenie hub, so this isn't the case.

[tonbut]

@gpbenton you're a saviour. I've finally got something that get the Pi to talk to the MIHO013 - engMQTTClient. Got the identify command working. Now to find what's different in the python code. Thanks for your patience so far.

[tonbut]

I've finally got this working as expected. It turned out that all the extra debug I'd put in slowed down my old pi too much and I think it must have missed the timing on the receive window of the MIHO013.

In tracking this down I went through all the code down to the radio level and understand it much better! One difference I notice @whaleygeek is that the C code uses a random int as bytes 4 and 5 and your python code uses a the constant encryptPIP. This appears to make no difference to behaviour but maybe to security?

I've placed a queue into the MIHO013 class which will now send a single message if one is available each time a message is received. This is working. Mapping this to the action methods on the interface is easy.

However I have a stylistic question on the requesting of battery level and diagnostics: What I think would make sense is to specify refresh period with defaults for automatically updating the values. I.e. maybe every 1 hour or 6 hours a request for these levels is put into the queue automatically and then the internal readings state will be updated and available when, for example, when get_battery_voltage() is called.

Unfortunately at the moment I see no way of reliably obtaining the pipe temperature or valve position. The depth of the diagnostic flags is really nice touch though.

Can anyone give any guidance on what low power mode does?

Hopefully when this work is done it can be merged back into the main repo.

[gpbenton]

Can anyone give any guidance on what low power mode does?

Here is the answer I got from Energenie technical support

2. In answer to your question on the Low Power mode the intention with this function is to enhance battery life by limiting the hunting of the actuator, ie it limits small adjustments to degree of opening, when the room temperature is close to the set point. A consequence of the Low Power mode is that it may cause larger errors in controlling room temperature to the set temperature.

In practice, I don't find it works very well - the temperature varies too much.

[devanubis]

Hi @tonbut, did you ever push your working MIHO013 queue code?

I've got some free time coming up this month (well, cutting back on overtime) and I'd like to try getting temperature logging and TRV controls running.

Thanks to everyone who's had the time to work on this so far BTW

[tonbut]

I've made pretty good progress on MIHO013 code. I've got all the main messages working except for the setTemperature command. This appears to be getting ignored by the eTVR. Currently I'm working around this by using the setValve open/close commands. I haven't commited anything yet as I was hoping to try again for a last push... I never cloned the original repo to start working, just downloaded a zip. Let me take a look at the state of things and I'll get back to you. I have had my code running for a month or so now with no issues so it can't be all bad.

[tonbut]

I've cloned this repository at: https://github.com/tonbut/pyenergenie and commit the changes to three files. The major change is that the MIHO013 class in Devices.py has a number of new implemented methods:

• def get_battery_voltage(self)
• def get_ambient_temperature(self):
• def get_diagnostics(self):
• def get_setpoint_temperature(self):
• def set_setpoint_temperature(self, temperature):
• def set_valve_position(self, position):
• def set_identify(self):
• def turn_on(self):
• def turn_off(self):
• def enable_thermostat(self):

By default voltage and diagnostics are requested once per hour asynchronously. When the code is first run and the first message is received from an MIHO013 the voltage and diagnostics are both requested immediately. However because of the slow nature and the fact that they don't listen continuously it can be 5 minutes before the corresponding methods return anything other than None.

Hope this works for you. Let me know how you get on.

[Quanghoster]

Hi, Just discovered this repository as I've purchased an ENER314-RT to look into building an addon for Hass.io (Home Assistant) for eTRVs in particular. I am looking to develop the addon to be able to create automations that link the eTRV's back to hassio so that they can be used to generate events which can be used to turn on the boiler through hive. I would be interested to know if anyone else has been working towards this? I think this library will make my task a lot easier :-)

[Mark-C-uk]

Id look at smartthings. Someone has wrote a script to pull the information then automations are in ST. You would just need to check if there is anything down to integrate hive

On 13 Jan 2018 15:10, "Andy Dennis" notifications@github.com wrote:

Hi, Just discovered this repository as I've purchased an ENER314-RT to look into building an addon for Hass.io (Home Assistant) for eTRVs in particular. I am looking to develop the addon to be able to create automations that link the eTRV's back to hassio so that they can be used to generate events which can be used to turn on the boiler through hive. I would be interested to know if anyone else has been working towards this? I think this library will make my task a lot easier :-)

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/whaleygeek/pyenergenie/issues/12#issuecomment-357442211, or mute the thread https://github.com/notifications/unsubscribe-auth/APmCZULE5Oai00bORJj3dYtb8sJ23iz3ks5tKMdsgaJpZM4GNbVu .

[gpbenton]

I would be interested to know if anyone else has been working towards this? I think this library will make my task a lot easier :-)

Since this thread was started I have been using my MQTT driver with HA quite successfully. The advantage being that HA will not have to run on the same machine as the ENER314-RT.

[Quanghoster]

@gpbenton thanks - that looks interesting. I've a dedicated Pi3 for HA which I would think can handle it, but good to have the flexibility to offload if necessary. I can run it up on a separate Pi to test it out and maybe look at developing an HA add-on to include it at some point. I'll take a look at your project. :-)

[wuftymerguftyguff]

I fancy adding this. Fork, fix and raise a pull? Or is there other work afoot?

[whaleygeek]

@wuftymerguftyguff I'm not specifically working on this part at the moment, and I don't have the right plumbing to use eTRV's in my house. Feel free to fork&fix, all contributions are welcomed.

Sometimes it's not always possible for me to merge a PR if I don't have a simple way to test at this end, but I do leave useful PR's active (and sometimes update the README to refer to other work where it is relevant)

[tonbut]

If it's the eTVR support you want rather than HA integration then my fork at https://github.com/tonbut/pyenergenie has been up and running stably for nearly two years now.

[wuftymerguftyguff]

Well if Tony has already done the work how can I facilitate testing his fork / PR?

[beyond-code-github]

I've got all the main messages working except for the setTemperature command. This appears to be getting ignored by the eTVR

@tonbut Great to see that this thread is still kinda active, I've just bought some of these eTRVs and am going to give your branch code a whirl over the next couple of weeks. I'm curious though, did you ever resolve the issue above?

[whaleygeek]

The eTRV's are a bit fiddly, because the device spends a lot of it's time in sleep mode, to conserve battery power. So getting messages to it and responses back requires a bit of knowledge about it's awake/sleep cycles, and also careful scheduling at the host end.

[tonbut]

Yes @beyond-code-github I think the temperature command is working now. I still have this stuff in operation but mostly I use the binary turn_on turn/off. Having the temperature reading so close to the ground on the radiators doesn't give the best readings for me, at least in my house, so I tend to rely on other temperature sensors. As @whaleygeek says though, they are a real pain to debug and know you've got them working properly as they only listen for commands for a millisecond or so every five minutes when they wake up to send a reading. The set_identify command is useful for knowing they are actually working and listening as this command makes the LED flash for while.

[beyond-code-github]

Thanks for the responses folks. So in terms of successfully dispatching messages, is the only way to do this to essentially spam it for 5 mins or so until querying the result indicates it has taken effect?

[tonbut]

No, the way it works is that the commands are queued up. Then when a message is received from a eTVR, the code quickly sends a queued message straight out. When it receives a another response it sends the next one. This works without spamming. Spamming would be unlikely to work because the window where the listening happens is so short!

Knowing if the result takes effect or not is hard though - because I see no messages you can sent to the eTVR that allow you to query its current valve state. You can query temperature, battery level, and diagnostic flags, that's all.

[beyond-code-github]

@tonbut Ahhaaa thanks for the clarification, I see now that you've actually written the queue mechnism into the device code itself, that's awesome. Is there literally only enough time to send one message then?

I can see why using commands to have the valve fully open or closed can be beneficial as at least you can check that! Looking forward to having a play with this when I have some time.

[tonbut]

Only one message at a time, but for each message sent another reply comes back. So as long as it goes well they chain together.

[beyond-code-github]

Ahhh ok that's good to know, makes things a little more manageable

[beyond-code-github]

I'm intruiged by this, was there a reason for the multiplication by 8? https://github.com/tonbut/pyenergenie/blob/master/src/energenie/Devices.py#L1090

[tonbut]

I think it's just because the underlying data structure uses integer values that increments in 1/8th degree steps.

[beyond-code-github]

Is anyone here any good at reading the original C code provided by energenie?(https://energenie4u.co.uk/catalogue/download_software/ENER314-RT%20programs.zip) I was looking deeper into the set temperature command after reading the blog post mentioned in this thread, and I think the current command being used might be wrong. The params featured in the eTRV C code are:

#define PARAM_JOIN_RESP     0x6A
#define PARAM_JOIN_CMD      0xEA
#define PARAM_POWER         0x70
#define PARAM_REACTIVE_P            0x71

#define PARAM_CURRENT       0x69
#define PARAM_ACTUATE_SW    0xF3
#define PARAM_FREQUENCY     0x66
#define PARAM_TEST          0xAA
#define PARAM_SW_STATE      0x73
#define PARAM_TEMP_SET      0xf4

#define PARAM_VOLTAGE       0x76
#define PARAM_TEMP_REPORT   0x74

Most of these map well to the parameters in the python lib, but PARAM_TEMP_SET 0xf4 is not mentioned anywhere, instead the set command is using 0x74 which is actually the param used to fetch a reading - https://github.com/tonbut/pyenergenie/blob/master/src/energenie/Devices.py#L180 https://github.com/tonbut/pyenergenie/blob/master/src/energenie/OpenThings.py#L89

So now I'm currently trying to figure out how to map the parameters passed in the C code onto a message structure in the python lib, particularly the two data fields:

HRF_send_FSK_msg(HRF_make_FSK_msg(manufacturerId, encryptId, productId, sensorId,
                   4, PARAM_TEMP_SET, 0x92, (data & 0xff), (data >> 8 & 0xff)), encryptId);

Here is the method involved:

uint8_t* HRF_make_FSK_msg(uint8_t manufacturerId, uint8_t encryptionId,
    uint8_t productId, uint32_t sensorId, uint8_t paramNum, ...){
    uint8_t *msgData = (uint8_t*)malloc(MAX_FIFO_SIZE * sizeof(uint8_t));
    uint8_t i;
    va_list valist;
    //                                      msgData[0] reserved, reg used while sending
    msgData[MSG_REMAINING_LEN+1] = MSG_OVERHEAD_LEN + paramNum;
    msgData[MSG_MANUF_ID+1] = manufacturerId;
    msgData[MSG_PRODUCT_ID+1] = productId;
    msgData[MSG_RESERVED_HI+1] = rand();
    msgData[MSG_RESERVED_LO+1] = rand();
    msgData[MSG_SENSOR_ID_2+1] = (sensorId >> 16) & 0xff;
    msgData[MSG_SENSOR_ID_1+1] = (sensorId >> 8) & 0xff;
    msgData[MSG_SENSOR_ID_0+1] = sensorId & 0xFF;

    va_start(valist, paramNum);
    for (i = 0; i < paramNum; ++i)
    {
        msgData[MSG_DATA_START+1 + i] = va_arg(valist, uint);
    }
    va_end(valist);
    setupCrc(msgData + 1);
    encryptMsg(encryptionId, msgData + 1, msgData[MSG_REMAINING_LEN+1]);
    return msgData;
}

The message formats in the python lib feature 'length' and 'data' properties. I'm fairly certain the data property is just going to be the decimal temp value according to http://blog.energenie4u.co.uk/tech-gadgets/something-weekend-two-way-raspberry-pi-transceiver-board-etrv-guide/ but I don't know about the length?

Any ideas?

[tonbut]

You might be onto something here! It's a long time since I worked on this code and I don't remember where I dug up the code to set the temperature but it was likely one of the C examples as @whaleygeek suggested. Maybe I got it wrong for this command or maybe there are conflicting examples.

Don't worry about the length. I think if you write the correct temp value into the structure as per my code then the Openthings encoder does the magic as long as the metadata for the structure is set up correctly. I don't remember how this works though now - i'd need to refresh my memory. Have a look at OpenThings.py file.

[tonbut]

Oh one thing I do remember regarding the 0xF4 vs 0x74 - commands (outbound) have to bit set, reports (inbound) have bit cleared - the encoder/decoder does this automatically. So it doesn't matter which is used.

[beyond-code-github]

Ahhhh ok, I can see this line in OpenThings.py is this what you're referring to?

payload.append(0x80 + paramid) # WRITE

Would this not just do an addition rather than setting the first bit directly?