Simple script to stream OpenThings messages via stdout/in

[jollytoad]

Hi David, this is great project btw. I'm hoping to try to use it to enable my home heating system to be completely self-contained, at present I'm using the MiHome REST APIs to read and control my eTRVs. My project is a node.js system... https://github.com/jollytoad/miheating - it monitors the eTRVs and fires my boiler when an eTRV is demanding heat. It's current running on Raspberry PI, and I also have the Energenie PI board attached - although it's not in use yet. As my project is node.js, I'd like to just run a separate Python script from it feeding the OpenThings messages (in JSON representation) via stdout/stdin using your code.

I tried to hack together a small script like this:

import energenie

energenie.init()

def incoming(address, message):
    print(message)

energenie.fsk_router.when_incoming(incoming)

while True:
    energenie.loop()

but I'm finding that it outputs more than just the raw JSON messages, and I can't work out what I'd need to do to read message from stdin and transmit them. I'd also like to output only registered devices (eg. only my eTRVs, not my power adapters). Although I'm a software dev by trade, unfortunately my Python-fu is pretty much non-existent.

Any suggestions would be great, cheers - Mark.

[whaleygeek]

Hi Mark,

Thanks for the feedback - interesting idea using stdin/stdout (presumably you want to use process stdstream redirection from node.js).

There are probably one or two left over print() statements from earlier debugging exercises - mostly I wrap these in a trace() which can be turned off on a per module basis. So a search for print() and trace() and even warning() through the code could find them all, and you could just comment them out.

To read in a message from stdin and send it, well, the problem there is that you need to be reading stdin at the same time as processing incoming messages, so you'd need a couple of threads to do that (as cooperatively reading stdin is possible but a bit tricky to get right).

To process just eTRV messages, you could filter incoming messages on one of the header fields, which contains the productid of the device - so you could filter only the eTRV device type.

Another example to look at would be the energy monitor example here: https://github.com/whaleygeek/pyenergenie/blob/master/src/mihome_energy_monitor.py and look at the call to Logger.logMessage() as that only processes actual messages to generate a CSV file. But I think you'd be better off just stripping out any spurious print() or trace() or warning() like this

def trace(msg):
   pass # print(str(msg)))

I'll have a think about how you could read from stdin - it might be that #53 and #57 might later surface some work that could be useful to you.

[whaleygeek]

Also when I finally get round to looking at NodeRed in #38 I'm sure I will come across all the same issues there that you are looking at at the moment, so keep an eye on that one too.

[jollytoad]

Thanks for the advice David, I got a simple script to dump etrv reading to stdout now, suppressing other print statements. I'm thinking that I could just avoid any threading issues by having a separate script for transmitting messages, does the code allow this, or will the radio device be locked to the one running instance?

[whaleygeek]

Hmm, there is only one physical radio, and it uses the SPI port, driven under the bonnet by GPIO reads and writes, and it is half duplex. You can't be transmitting when you are receiving, and the radio has to be reconfigured between transmit and receive by sending different SPI commands. You're bound to bump into some mucky mutually exclusive access issues at the raw hardware layer.

At the gpio layer I do open the gpio peripheral in shared mode here:

https://github.com/whaleygeek/pyenergenie/blob/master/src/energenie/drv/gpio_rpi.c#L88

Although without any form of coordinated synchronisation, this would just end in tears to be honest.

If you had a lock object between two processes, then you might as well just deal with the issue with two threads, as it would be easier to communicate between the two independent programs if they are in the same memory space.

I do have some work planned in #9 to build a message scheduler where it schedules tx and rx slots in separate timeslots that would resolve this issue without any threads, as that would be fully cooperative. I've architected the inner part of the transmit and receive pipelines to allow for this work in the future.

So I think your best bet would be to approach it from a cooperatively scheduled angle with the code in a single thread in a single process, as that would avoid any mutually exclusive hardware access issues.

energenie.loop() configures the radio into receive mode for a fixed duration (which you can override with a parameter), and when it is not receiving, you can be transmitting by calling any of the MIHO013 methods here, which will OpenThings encode the messages for you:

https://github.com/whaleygeek/pyenergenie/blob/master/src/energenie/Devices.py#L892

If you want to transmit a message to your eTRV you can do it like this: https://github.com/whaleygeek/pyenergenie/blob/master/src/control_any_noreg.py#L22

As you won't be in an energenie.loop() at that point, the message will be transmitted fine.

Your only remaining problem then is to read commands from stdin such that your main thread does not block. There are ways to do that, that are well documented (I found a number on StackOverflow when searching a while ago for example).

Note that the line number references in this issue are referenced to HEAD, so will probably change when I commit new code, but that won't be for a while yet.

So, in summary, I think you would be best to run it all cooperatively, in a single process, and look for a way to read stdin either in a separate thread, or cooperatively. That will be something you could easily research elsewhere and find a standard solution to, without needing to use the pyenergenie code any different to any of the existing examples.

Hope this helps.

[whaleygeek]

For example, you know this is running on a Pi, so this should work using select()

https://repolinux.wordpress.com/2012/10/09/non-blocking-read-from-stdin-in-python/

[whaleygeek]

Ok, so here is a self contained example showing how you can write a Python thread to independently read from the stdin stream. At the end of each input line, it puts the line in a queue, which you can then cooperatively poll in your main loop, and peel off one message at a time and process it accordingly. This will allow your energenie.loop() to still receive messages. The q.get() will read one line at a time when it is available, so you'll have to process that line and turn it into an appropriate function call to your eTRV MIHO013 object, to trigger it to transmit the message you want.

try:
  import thread
except ImportError:
  import _thread as thread

try:
  readin = raw_input
except NameError:
  readin = input

try:
  import Queue
except ImportError:
  import queue as Queue

import time

q = Queue.Queue()

# poller thread
def poller():  
  while True:
    data = readin()
    q.put(data)

in_thread = thread.start_new_thread(poller, ())

# main thread
while True:
  if not q.empty():
    data = q.get()
    print(data)

  print('sleep 1')
  time.sleep(1)

[whaleygeek]

Note I haven't implemented all messages types for eTRV's yet, #28 and #12 will probably mop those up. I have an eTRV here, but haven't had time to test all the messages out yet, so there might be a bit of 'finishing off' of the MIHO013 class required, depending on what you are trying to achieve.

[pvanderlinden]

@jollytoad Did you manage to get more working from the MIHO013? I just bought 2 of them only to discover I can't seem to control them from Raspberry PI, I can "pair" and read the Temperature which is broadcasted every 5 minutes, but all other messages seems to be ignored. The response from Energenie is very unhelpful so far, stating the device is "too complicated" for the raspberry pi.

[whaleygeek]

As per my message on Ed's pull request, the "unfinished business" is here:

https://github.com/whaleygeek/pyenergenie/blob/master/src/energenie/Devices.py#L892

If you were eager to get going with something, you could implement the handle_message in the MIHO013() class in a similar manner to that in the other classes in this file, and route the decoded messages through to the appropriate handlers. That way calling the appropriate get_xxx() method would indeed return something useful.

A starting point would be to just dump the json in handle_message via a print message, as that will give you a very good indication of the data that is coming out. Then use similar techniques to the other classes in that file to crack open the various data fields.

Sorry if that is not a complete answer. This is on my radar to do more work on this, although as it is not easy for me to install these in my house, I've had perhaps less motivation than others to crack on with this.

[whaleygeek]

And this is what a typical handle_message looks like, once you have captured some representative received messages from the device:

https://github.com/whaleygeek/pyenergenie/blob/master/src/energenie/Devices.py#L966

Sending is a little harder, as I think the eTRV only opens up a receive window close to it's reporting period, so there is some additional scheduling required to queue up setpoint setting commands, I think.

[pvanderlinden]

@whaleygeek So far I'm having a difficult time getting anything out the device except pairing. I tried sending some of the messages from messages.txt (https://github.com/whaleygeek/pyenergenie/issues/21) with filled in values but no response from the device so far. One thing I keep noticing is the undecodable OpenThings messages every know and then as well, not sure if those come from the eTRV though. They have the correct length byte at the start, but can't be decoded.

[jollytoad]

My radio board seems to be completely non-functional atm, and I've not had time to investigate further. From my understanding of the eTRVs, you need to transmit as soon as you receive a message from it, as David says it has a very small receive window. I'd hacked about with a script that works at a much lower level, just waiting for eTRV messages and then transmitting, but for some reason the radio just doesn't initialise properly any more :(

[whaleygeek]

From what I know, the OpenThings message types and data types are as follows:

temperature report: paramid 0x74, typeid 0x92 (2 bytes data) - hi bit clear means a report set temperature cmd: paramid 0xf4, typeid 0x92 (2 bytes data) - hi bit set means a command

My tests show eTRV reports temp on roughly a 5 minute interval. Receive window then opens for a few hundred ms and closes automatically if nothing received. You can send a command in during that RX window. Some messages have acks, some do not.

So, from a reception point of view, you could just fill in the handle_message() in the MIHO0013 class to record the incoming 0x74 report, and then any time you do get_temp() in your python you would get the last known actual temperature reading. I'm not sure if this is pipe temperature or ambient temp though.

From a transmission point of view, the eTRV will only receive when the receive window is open, and it closes quite quickly, due to it being a battery operated device that wishes to conserve energy.

I'd probably do this by making the MIHO013.set_temperature store the requested value in a cache variable, and set a flag to say it has been updated. Then when the next temperature report comes in, if the new_setpoint flag is set, request an fsk transmit of a OpenThings message sending the setpoint temperature, and bail out quickly, getting the radio in FSK rx mode as soon as possible. Note there is a state machine in the radio module that allows you to auto-return to rx mode quickly. If you are already in rx and push a tx, when you pop, it returns you back to rx mode as fast as it can (without going via standby). So you can mostly leave the radio 'hot' all the time, ping ponging between tx and rx.

If the eTRV device received the message, it will then send an acknowledgement, which should be received and routed to the MIH013 instance for that device. The handle_message can then clear the new_setpoint flag, and all will be happy. If the device did not send an ack (presumably because it did not receive the message, or because the pi radio was not put into rx mode quick enough, or the reply collided with some report from some other device), the new_setpoint flag will still be set. The next time a report comes in from the eTRV, it will try again to send the setpoint. Eventually it will get through.

This also means that the MIHO013 class instance knows if there is a pending setpoint command or not, so you could surface that via the API so your app knows if the device is in sync or not, I suppose.

What I would say is, don't bank on a really fast and responsive system. Because the report interval is every 5 minutes, testing this is quite hard, unfortunately.

Hope this helps. Sorry to be so busy with other stuff at the moment!

[whaleygeek]

Also...

I had that problem with my radio board in the early days. What version of the code are you using? Some of the earlier versions of code (from energenie and the rewritten stuff from me) did not correctly pulse the hardware reset line, and it was possible to get the radio into a software state where it had locked up. All those problems went a way when I implemented proper reset.

In particular, carefully removing and refitting your board, or power cycling the Pi (yes, power cycle, not just reboot) can guarantee a hard reset to prove if this is the case.

The RFM69 SPI protocol has a few dark alleys in it, and it is possible to get the radio into a state where it looks like it is not responding.

[whaleygeek]

If you suspect reset issues (unlikely in the latest code, but hey...) you could modify this script: https://github.com/whaleygeek/pyenergenie/blob/master/src/cleanup_GPIO.py to set reset as an output and pulse the reset line to be sure. Although the reset in the latest code is done here:

https://github.com/whaleygeek/pyenergenie/blob/master/src/energenie/drv/radio.c#L250

150ms is more than long enough for reset I think.

energenie.init() calls radio.init()

https://github.com/whaleygeek/pyenergenie/blob/master/src/energenie/__init__.py#L36

which then calls radio_reset:

https://github.com/whaleygeek/pyenergenie/blob/master/src/energenie/drv/radio.c#L277

I did find that when I had a locked up radio and no working reset, I really did have to power cycle the Pi (reboot not being enough)

[pvanderlinden]

I still only manage to do the pairing, for the rest I get no response from the valve at all (except the temperature every 5 minutes). I send the message out within a few ms, and it is send within a few hundred.

('in', 1475584830.512824)
Incoming from (4, 3, 1242): {'header': {'sensorid': 1242, 'productid': 3, 'encryptPIP': 6468, 'mfrid': 4}, 'type': 'OK', 'rxtimestamp': 1475584830.511964, 'recs': [{'paramunit': 'C', 'typeid': 144, 'valuebytes': [24, 205], 'value': 24.80078125, 'length': 2, 'wr': False, 'paramname': 'TEMPERATURE', 'paramid': 116}]}
('sending', 1475584830.513603)
Send: {'header': {'sensorid': 1242, 'productid': 3, 'encryptPIP': 39478, 'mfrid': 4}, 'recs': [{'typeid': 0, 'length': 0, 'paramid': 116, 'wr': False}]}
('send', 1475584830.71937)

Using the latest code (master) btw. I Also tried a lot of different param id's as well (exercise, battery level).

[whaleygeek]

Hmm, that send data looks wrong though, the recs has typeid 0 length 0 paramid 116 wr false.

If you are trying to set the setpoint, you need typeid 0x92 (142) paramid 0xf4 (244) length 2 with a value of say 24.800 (value bytes [24,205]) to set the same setpoint. This should then send an ack back of a similar message, echoing your new setpoint, I think.

The SWITCH message sending is a good template to look at how the sending works, you have to use OpenThings.message() with a message template, and override the fields with your actual values. Here is a SWITCH message being constructed and sent in the turn_on() method (which I know works)

    payload = OpenThings.Message(SWITCH)
    payload.set(header_productid=self.product_id,
                header_sensorid=self.device_id,
                recs_SWITCH_STATE_value=False)
    self.send_message(payload)

Here is the template message that uses:

SWITCH = { "header": { "mfrid": MFRID_ENERGENIE, "productid": PRODUCTID_MIHO005, "encryptPIP": CRYPT_PIP, "sensorid": 0 # FILL IN }, "recs": [ { "wr": True, "paramid": OpenThings.PARAM_SWITCH_STATE, "typeid": OpenThings.Value.UINT, "length": 1, "value": 0 # FILL IN } ] }

So, I would expect your send() dump to show the paramid and typeid correct, the length as 2 (because there are two bytes in the payload), and the wr bit set true (because the high bit of the paramid is set, indicating a command).

[pvanderlinden]

I was trying to get settings and statusses from the device, I only tried getting the set temperature point, the battery level and the exercise command in different ways.

[whaleygeek]

Oh I see, you're trying to read paramid 116 (0x74).

You might have to provide it with the correct type id 0x92 which matches what the report says. It depends how the firmware at the receiving end is coded, but it is possible that it will ignore commands with a typeid that does not match what it would normally report.

Given it is a tiny micro controller in the eTRV, I wouldn't be surprised if it looks for a command starting 0x94 0x72 and rejects anything that does not match.

[whaleygeek]

I'll have to check the OpenThings spec (was called OpenHEMS), to see what they say about read requests, as to whether there needs to be a typeid or not, I can't remember what the rules of the protocol are.

You can request a free copy of the protocol spec from Sentec here: http://www.o-things.com/

[jollytoad]

This is as far as I got with my script (to feed etrv messages via stdin/stdio)... https://github.com/jollytoad/pyenergenie/blob/master/src/etrv_stream.py

NOTE: Example message in a comment at the bottom of the file - based on the message send out by the official MiHome gateway.

but haven't had chance to fully test yet due to the radio problems I mentioned. Thanks for the advice on that David - I'll give it a go whenever I get time to revisit this.

[pvanderlinden]

I have tried a set of types as well: types=[0x92, OpenThings.Value.UINT, OpenThings.Value.UINT_BP16, OpenThings.Value.SINT_BP16] I have cycled through them, but still no response.

I have a copy of the OpenThings Protocol spec, unfortunately that is missing the Energenie specifics ofcourse.

[whaleygeek]

Sorry, I mean SINT_BP8 - typeid 0x9x that looks to be in the message received, is SINT_BP8 (which means it has a binary point at position 8, so 8 bits of binary precision) - that's why the number returned always looks very screwy with the decimal part, as it's a decimal approximation of the binary representation). The 2 in the 0x92 means that it has been encoded into two bytes (so the high byte will be the whole part, and the low byte will be the fractional part).

It's only a hunch, but I think the embedded microcontroller will probably be quite selective about what it will accept.

[pvanderlinden]

I tried that as well now. Still no responses from the device. Did anyone manage to find eTRV_Menu as mentioned in http://blog.energenie4u.co.uk/tech-gadgets/something-weekend-two-way-raspberry-pi-transceiver-board-etrv-guide/ ?

[whaleygeek]

Hi, I hadn't realised that program existed (and unfortunately Energenie seem to have forgotten to mention it to me too! oops!)

I'll have a look through when I get a chance, this is bound to have the necessary information we need to get the eTRV's working reliably.

[pvanderlinden]

Did you find the program? It is mentioned in the blog post, but I haven't been able to find it.

[jollytoad]

@pvanderlinden It's missing from the download on the blog, I asked support a while ago and they sent me a zip containing the files, i'll find somewhere to upload it to...

https://drive.google.com/file/d/0B7Jhx_Ijf3d7TS1tSzBpcEtfeGM/view?usp=sharing

[gpbenton]

I have been communicating with my two eTRVs using my engMQTTClient since January.

I haven't had the heating on for about six months, but I have heard the valves being exercised periodically, and the temperature is reported, so I'm sure its still working.

[pvanderlinden]

Thanks @gpbenton I will have a look at that as well. @jollytoad Thanks!

With the source of eTrv_Menu I got it to exercise and to send it the voltage. Going to have a better look tonight/this weekend

[pvanderlinden]

I did some more testing, but it is really hard to get the message delivered to the eTRV. Maybe it is the delay between receiving and sending.

@whaleygeek I noticed you have a script to compile the radio for mac. Did you manage to get an radio for a MAC/PC some where? That would help a lot with testing things.

[whaleygeek]

@pvanderlinden the mac build script was really as a way to 'get ahead' with resolving compile time errors when I was mobile, with a view to being 'more ready' when I was back home and able to test on the Pi. However, there is some work going on over here that will eventually aim to support such a board over USB on Mac and PC...

https://github.com/RyanteckLTD/RTk

This is a fork of some code I wrote a couple of years to allow me to develop the hardware chapters for a book I was writing at the time, in a platform independent way. I think Ryan plans to support SPI on this board eventually, in which case it would be possible to saw-off the pyenergenie software at an appropriate point and remote the SPI communications via the RTk.GPIO board perhaps.

I haven't tried this, but if you press the join button on the eTRV, I think it wakes up the device and immediately sends a JOIN_REQ and goes into receive mode to wait for the JOIN_ACK. This might be a way to speed up your testing of different message types, at least?

[whaleygeek]

There are also some relevant side discussions about eTRV happening on this (not processed) pull request from Ed:

https://github.com/whaleygeek/pyenergenie/pull/17

I don't plan to merge that pull request though, as it is based on a very old version of the code, and a lot has changed architecturally since Ed did that work.

[pvanderlinden]

I got a lot further yesterday evening and this morning, and got a lot working now. Some conclusions, I'm 99% sure of:

• You can't set the reporting period below 5 minutes
• The device doesn't accept messages with multiple records
• The device ignores multiple messages in sequence
• The device doesn't ACK on messages, making it difficult to see if a message was accepted
• You can't get the current state of different settings (valve state, temperture setting, etc)

[gpbenton]

I haven't been able to do any of these things either. The diagnostics response does give some information about its current state, but nothing particularly useful, and I discovered yesterday that it doesn't detect sticking valves very well either.

[pvanderlinden]

Just some more progress here.

• I think there were 2 small bugs causing some problems, will test it later and raise a pull request
• I think I got most things working, including an interesting feature: You can send the temperature, so it uses that instead of it's internal readings (which solves the problem of the bad readings you get: close to the radiator and windows).

I'm going to work a bit more on it, and raise some pull requests for it.

[pvanderlinden]

I hope I can soonish have some pull requests for this.

I got some more information from energenie: The valve only listens for 200ms after sending a temperature report. Diagnostic report byte:

• d0, motor current below expectation * (not implemented)
• d1, motor current always high * (not implemented)
• d2, motor taking too long to open/close *
• d3, discrepancy between air and pipe sensors * (not implemented)
• d4, air sensor out of expected range *
• d5, pipe sensor out of expected range *
• d6, low power mode is enabled (default is disabled)
• d7, no target temperature has been set by host
• Data high byte:
• d0, valve may be sticking *
• d1, valve exercise was successful *
• d2, valve exercise was unsuccessful *
• d3, driver micro has suffered a watchdog reset and needs data refresh *
• d4, driver micro has suffered a noise reset and needs data refresh *
• d5, battery voltage has fallen below 2.2V and valve has been opened *
• d6, request for heat messaging is enabled (default is disabled)
• d7, request for heat

@whaleygeek Not sure how familiar you are with RF69, but I'm trying to build an USB tranceiver based on the feather (https://www.adafruit.com/product/3177). Got it to receive, but not to send (I can't read it through the raspberry pi): https://github.com/pvanderlinden/feather_energenie

[whaleygeek]

Hi @pvanderlinden

That info looks like internal communications within the eTRV, I'm not sure if any of that surfaces to the outside radio (as it doesn't fit within the OpenThings protocol format that it uses). It does however indicate the sorts of things that might be available if they surface via OpenThings.

Yes, reasonably familiar with RFM69, I wrote the C driver in this project from the ground up from the data-sheet. I had seen the feather product, someone else pointed me at that a while ago in one of the other issues on this repo I think, might be quite nice way to get access via a PC/Mac/Pi perhaps.

[whaleygeek]

P.S. @pvanderlinden all those serial.print's in the .ino code will significantly slow down the loading of the fifo when in transmit mode, and may cause the fifo to underflow - depending on the FIFO start condition you set in the appropriate flag register, you may be in a condition where the RF69 sequences into the transmit state and starts transmitting data in the FIFO, but your arduino code is too busy servicing the serial port with all that debug, that it doesn't get bytes in the FIFO quick enough.

I'd suggest you comment out all the Serial.print's and try again. Also check what condition you trigger the txstart condition, as once you trigger start, you're basically timing critical from then on.

The transmit configures the payload data to come out on one of the GPIO pins of the RF69, so shove a scope or logic analyser on that pin and see if you have any unexpected gaps in the payload data.

[whaleygeek]

In fact you won't have a gap, you'll get a fifounderflow interrupt flag set, and it will probably stop transmitting at that point and you'll get a short payload.

Especially seeing that the 'times' parameter allows for payload repeats, once your FIFO start condition is met, you have to sustain a minimum data transfer rate over the SPI that is at least a bit faster than the bitrate you have configured the radio (which I recall is 4800bps in this system).

[pvanderlinden]

@whaleygeek according to energenie those 2 bytes are returned when request diagnostics ( https://github.com/pvanderlinden/pyenergenie/blob/hack/src/send_on_recv.py#L93 ).

I have removed the printing to serial which didn't help unfortunately. I also tried the other way of sending (putting the radio in standby, then fill the fifo, then put the radio into transmit mode), where I got the packet sent interrupt after each message, but still didn't see anything on the receiver of the raspberry pi.

I suspect I'm doing something simple wrong (at the start I didn't receive anything either because I misconfigured the radio), it's just the question how to find out what I'm doing wrong.

[whaleygeek]

For issues related to eTRV support, can we all start conversing on issue #12 from now on, as there are a few people interested in this work.

[whaleygeek]

Sorry this seems to be a very old issue now, I have captured a number of the ideas here. If you have a specific problem remaining with the eTRV, please do log it as there are a number of forks around now that seem to have the eTRV working