Implement ZWave Binding

[GoogleCodeExporter]

General ZWave Support

Implement binding for ZWave Controller/Device/Network support. 

Original issue reported on code.google.com by bmcro...@gmail.com on 29 Apr 2013 at 12:25

[GoogleCodeExporter]

Sounds good. I still have to figure out how to add you so we both can work
off the same codeset.

I will be working on validation of item types to command classes. Then just
more robust error handling and zwave network handling .

Original comment by bmcro...@gmail.com on 16 Jul 2013 at 6:19

[GoogleCodeExporter]

You should be able to assign me the committer role in the permissions 
configuration of the clone. I've completed the dimmer handling and (de)muxing 
of multi-channel commands. Would be nice to push. Otherwise I can create a 
clone and you can pull from there.

Original comment by jwsp...@gmail.com on 16 Jul 2013 at 8:28

[GoogleCodeExporter]

i am afraid adding committers to clones is not possible but maybe i am wrong?!

Original comment by teichsta on 16 Jul 2013 at 8:33

[GoogleCodeExporter]

It seems so, I just read something like that. I'm afraid I will have to create 
my own clone and then you can pull / cherry pick commits from me...

Original comment by jwsp...@gmail.com on 16 Jul 2013 at 8:50

[GoogleCodeExporter]

I've created both a 1.2.0 and a 1.3.0 snapshot binary with the dimmer changes 
and the multi-channel binding. NOTE: the binding configuration has changed to 
this format:

zwave=nodeId:endpoint:command

endpoint can be omitted, it defaults to 1. If you specify command you also have 
to specify an endpoint. So you get for instance:

String ZwaveNode01HomeID    "Home ID [%s]" (gZwaveNode01) {zwave="1:1:homeid"}
String ZwaveNode01NetworkID "Network ID [%s]" (gZwaveNode01) 
{zwave="1:1:nodeid"}

Dimmer Light_LivingRoom_Dimmer "Living Room Dimmer [%d %%]" (GF_Living) 
{zwave="3"} 
Dimmer Light_Corridor_Dimmer "Corridor Dimmer [%d %%]" (GF_Corridor) {zwave="6"}

Switch Mech_Vent "Mechanical Ventilation low/middle." (GF_Kitchen) {zwave="9:1"}
Switch Mech_Vent_High "Mechanical Ventilation low/high." (GF_Kitchen) 
{zwave="9:2"}

It's rather rudimentary now. Only multichannel works, no multiinstance (old 
version).
There is no detection of multichannel capabilities yet, so it all depends on 
the right bindings in the items file. I'm able to succesfully control a fibary 
FGS 211 dual relay switch now, both switches and buttons.

Original comment by jwsp...@gmail.com on 16 Jul 2013 at 10:15

Attachments:

• org.openhab.binding.zwave-1.2.0-SNAPSHOT.jar
• org.openhab.binding.zwave-1.3.0-SNAPSHOT.jar

[GoogleCodeExporter]

Tomorrow I will create a clone, cleanup some stuff and push the changes.

Original comment by jwsp...@gmail.com on 16 Jul 2013 at 10:17

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Awesome - I can confirm my multi-switch is now fully operational with your 
latest binary. Many thanks!

Original comment by ben.jone...@gmail.com on 16 Jul 2013 at 10:48

[GoogleCodeExporter]

Could I put a request to change the ZWave binding logging? Currently there is a 
lot of 'debug' information being logged as 'INFO'. Would be good to update this 
to stop the main openhab logs getting saturated with ZWave messages. Easy 
enough to enable the debug logging while testing.

Original comment by ben.jone...@gmail.com on 16 Jul 2013 at 11:22

[GoogleCodeExporter]

I've created a clone and committed my changes. You can star my repository to 
get notifications on changes.

Original comment by jwsp...@gmail.com on 17 Jul 2013 at 11:17

[GoogleCodeExporter]

Guys - came across this today (https://github.com/zgmnkv/zwave4j). It is a Java 
wrapper around the OpenZWave library. Since the OpenZWave project is quite 
advanced I was wondering if this sort of wrapper might make our lives a lot 
easier with the ZWave binding? Save us re-inventing the wheel? Thoughts?

Original comment by ben.jone...@gmail.com on 21 Jul 2013 at 11:42

[GoogleCodeExporter]

I don't think we are reinventing something here. Using OpenZWave with java 
wrapper will bring in nightmare of keeping all possible (I see 4 at least - 
Windows, Linux x86, Linux ARM and Mac) builds of openzwave and supplying them 
with openHAB...

Original comment by belovic...@gmail.com on 22 Jul 2013 at 6:58

[GoogleCodeExporter]

There has been a discussion about using Open-Zwave before: 
https://groups.google.com/forum/#!topic/openhab/F_pxHjNLUXE.

Linking another library and generating JNI bindings makes OpenHAB less 
portable. E.g. I'm running FreeBSD and I've gone through quite a lot of hoops 
to get it running. I had to port the serial port library and the bluetooth 
library to FreeBSD to be able to use OpenHAB. OpenHAB is written in Java, it 
should run on any Java capable platform. This is only possible if you minimize 
native dependencies.

The Z-wave binding is in a state where most low level communication is properly 
implemented. Sure, it needs some tidying, and there are a lot of commands and 
device classes that can be added. But you would have to add these device 
classes, and bindings into a zwave wrapper binding as well.

Original comment by jwsp...@gmail.com on 22 Jul 2013 at 9:40

[GoogleCodeExporter]

Good points chaps - I just came across it today and wondered if it might be 
useful. I can see what you are saying and completely agree.

Original comment by ben.jone...@gmail.com on 22 Jul 2013 at 9:46

[GoogleCodeExporter]

Guys - I am seeing my ZWave binding freeze/die intermittently (roughly every 
couple of days). I have only had INFO debugging enabled so far (as it is so 
verbose) and this didn't seem to give any indication of a problem so I have 
configured DEBUG logging in a separate log and restarted openHAB. I am running 
the latest (from yesterday) build of 1.3.0 and using jwspuij's binding posted 
on Jul 16. I will keep an eye on it and see if I can see what is going wrong. 
Have you guys seen anything like this?

Original comment by ben.jone...@gmail.com on 23 Jul 2013 at 11:07

[GoogleCodeExporter]

Ben:

Uncanny timing. I was just about to post on this issue. I have noticed it
for a few weeks now; but as i didn't have a solution i was trying to hold
out to post. The only the only thing i have been able to find so far is the
lack of error handling on the zwave network stack. I noticed this morning
that my binding had gone belly up again, and i also noticed that i had to
CAN packet. I do not know for sure if this is the reason; but it is the
only thing i have found thus far that correlates. I will be looking at my
logs in detail tonight after work and will post what i find.

In the mean time, can you please implement the Statistics for your
controller node. What we need to look for is if you are getting any
NAK/OOF/etc packets as well. If you are getting a error stat, then my
theory is correct and what is happening is we receive a error from from the
network stack but we don't handle is gracefully so it kills the binding.
Which is something i knew we needed to look into, but I haven't figured out
how to extend the day past 24 hours yet to have time to work on it ;-)

Original comment by bmcro...@gmail.com on 23 Jul 2013 at 12:07

[GoogleCodeExporter]

I have the same issue, the only difference is that it dies every 6 hours on 
average. I will add statistics and see what I can do in the code.

Original comment by jwsp...@gmail.com on 23 Jul 2013 at 1:43

[GoogleCodeExporter]

Brian - I have just woken up to find my binding has again died. Checking my 
debug logs shows (like you) a CAN message arriving and after that the binding 
stops working. The refresh thread is still running but doesn't seem to be doing 
much. See below for the arrival of the CAN and then the 'dead' refresh loop. 
This continues until I restart. There are no other CAN packets received at any 
time in my log.

23:10:40.145 DEBUG o.o.b.z.i.p.SerialInterface$SerialInterfaceThread[:286]- Rx 
CAN
23:10:49.759 DEBUG o.o.b.z.i.ZWaveActiveBinding[:127]- Refresh Count: 1
23:10:59.759 DEBUG o.o.b.z.i.ZWaveActiveBinding[:127]- Refresh Count: 2
23:11:09.760 DEBUG o.o.b.z.i.ZWaveActiveBinding[:127]- Refresh Count: 3
23:11:19.760 DEBUG o.o.b.z.i.ZWaveActiveBinding[:127]- Refresh Count: 4
23:11:29.761 DEBUG o.o.b.z.i.ZWaveActiveBinding[:127]- Refresh Count: 5
23:11:39.761 DEBUG o.o.b.z.i.ZWaveActiveBinding[:127]- Refresh Count: 6
23:11:49.762 DEBUG o.o.b.z.i.ZWaveActiveBinding[:123]- Reseting Refresh Count 
to Zero
23:11:49.762 DEBUG o.o.b.z.i.p.SerialMessage[:66]- Creating empty message of 
class = 13, type = 00
23:11:49.762 DEBUG o.o.b.z.i.p.SerialInterface[:125]- Message placed on queue. 
Current Size = 2
23:11:49.762 DEBUG o.o.b.z.i.p.SerialMessage[:66]- Creating empty message of 
class = 13, type = 00
23:11:49.763 DEBUG o.o.b.z.i.p.SerialInterface[:125]- Message placed on queue. 
Current Size = 3
23:11:49.763 DEBUG o.o.b.z.i.p.SerialMessage[:66]- Creating empty message of 
class = 13, type = 00
23:11:49.763 DEBUG o.o.b.z.i.p.SerialInterface[:125]- Message placed on queue. 
Current Size = 4
23:11:59.763 DEBUG o.o.b.z.i.ZWaveActiveBinding[:127]- Refresh Count: 1
23:12:09.764 DEBUG o.o.b.z.i.ZWaveActiveBinding[:127]- Refresh Count: 2
23:12:19.764 DEBUG o.o.b.z.i.ZWaveActiveBinding[:127]- Refresh Count: 3
23:12:29.765 DEBUG o.o.b.z.i.ZWaveActiveBinding[:127]- Refresh Count: 4

Original comment by ben.jone...@gmail.com on 23 Jul 2013 at 8:39

[GoogleCodeExporter]

I've identified a couple of places where the serialInterfacethread  could die 
without giving a log message. Fixed those. I also implemented a watchdog timer 
that respawns the SerialInterfacethread in case it dies anyway. Furthermore 
I've changed a lot of log messages to the debug level to reduce the number of 
log messages during normal runs. Finally I've implemented graceful shutdown of 
ZwaveController and SerialInterface so that the by bundle can now be stopped 
and restarted using the equinox console.

It's running now on my install. If it's more stable I'll commit and upload it 
tomorrow.

Original comment by jwsp...@gmail.com on 23 Jul 2013 at 10:40

[GoogleCodeExporter]

Sounds great - look forward to testing.

Original comment by ben.jone...@gmail.com on 23 Jul 2013 at 10:41

[GoogleCodeExporter]

regarding to my email (Release 1.3): what it is your impression about the 
current binding code? When would it be ready for a first review? We'd love to 
announce the openHAB Z-Wave adoption ;-)

Btw: in order to make the binding available in the release we'd need an 
appropriate Wiki-Page to document all it's features. Any volunteers?

Original comment by teichsta on 23 Jul 2013 at 11:04

[GoogleCodeExporter]

I dont mind documenting.

Thanks everyone for the group effort on getting the Binding stable. Let us
know when you commit and push. I will pull them into my clone

Original comment by bmcro...@gmail.com on 23 Jul 2013 at 11:29

[GoogleCodeExporter]

I've committed and pushed the code. I'll try to build some new jars today.

Original comment by jwsp...@gmail.com on 24 Jul 2013 at 9:45

[GoogleCodeExporter]

Thomas, I'm not sure whether I'm qualified to talk about the state of the 
binding, since I've only made a couple of adjustments. From a technical point 
of view I'd say the following parts of the binding should be looked at, before 
releasing it to the general pulic:

- Queue handling. There is an output queue that queues messages, but incoming 
messages are handled on the serial thread. If parsing of the message and 
notifying items takes too long, we'll miss timeouts for the serial 
communication. IRL in my setup it does not seem to be a real problem though, 
but I cannot say how the zwave binding would handle 200+ zwave devices and 500+ 
item bindings for instance or on raspberry pi's.
- Error handling. A message that times out or gets a CAN / NAK is lost forever. 
Nodes are not declared dead after the network initialization is complete. On 3 
failed messages a node should be declared dead.
- Implementing CallBackId. Matching requests / responses is done now on the 
nodeId of the last sent message. In routed networks with many hops this will 
almost certainly fail as responses will arrive out of order. CallBackId can be 
used to do better matching, but we could also improve our own matching by 
maintaining more state than just the last sent nodeId.
- Battery operated slaves. I don't own them but i'm pretty sure that the 
binding cannot handle them. Maybe i'll buy something battery operated to 
implement this.
- The source code needs some tidying, we could do with removing dead code, more 
comments and javadoc.

Then from a user perspective:

- Documentation
- Implement more device classes. Dimmers and switches is not much. I can 
implement roller shutter, because I have one. I'll also get more devices like 
sensors and smoke detectors this year, but for this we really need more users / 
developers with more devices. We can try based on specs + openzwave code, but I 
prefer real devices to play with.

Original comment by jwsp...@gmail.com on 24 Jul 2013 at 10:46

[GoogleCodeExporter]

I've included both the 1.2.0 and 1.3.0 version again. Please test. I'm 
currently thinking that timing was the problem:

12:33:17.076 WARN  o.o.b.z.i.p.SerialInterface$SerialInterfaceThread[:322]- 
Unsolicited message received while waiting for ACK.
12:33:17.130 ERROR o.o.b.z.i.p.SerialInterface$SerialInterfaceThread[:363]- 
Message cancelled by controller for node 255

I get these entries in the log file now every now and then. What happens is 
that the controller gets a message from a node and puts it on the serial 
interface, while we are also sending a message to the controller. The result is 
a collision. We get an unsolicited message (and handle it) and subsequently a 
CAN.

In previous versions of the binding, the serial thread would register the CAN, 
but not reset the state that it is waiting for an answer. Hence it waited 
indefinitely for an ACK and not reset or timeout. No sending is done while 
waiting for an ACK.

I've added the reset of the waiting state on receiving NAK, CAN and also a 
timeout. Don't know whether these controller sticks support software / hardware 
flow control on the serial port. It could prevent this issue, but it's handled 
now anyway. But until retry is implemented we lose one message.

Can somebody confirm that the same messages appear in the log, but the binding 
continues?

Original comment by jwsp...@gmail.com on 24 Jul 2013 at 3:59

Attachments:

• org.openhab.binding.zwave-1.3.0-SNAPSHOT.jar
• org.openhab.binding.zwave-1.2.0-SNAPSHOT.jar

[GoogleCodeExporter]

So this is part of the error handling that i have mention we need to do for
the binding. Victor and I just wrote the binding to "work"; didn't handle
too many condition that could occur such as that above.

One of the most important things i think we need to tackle is setting up a
"Node Queue". What i mean is handle the messages for each node on their
own. SO that we can retry a to send messages; handle errors correctly; and
ensure there aren't collisions.

Original comment by bmcro...@gmail.com on 24 Jul 2013 at 5:17

[GoogleCodeExporter]

Ok - I am running your latest version jwspuij (what is your name btw?!) and 
will keep an eye out for these messages.

Original comment by ben.jone...@gmail.com on 24 Jul 2013 at 8:20

[GoogleCodeExporter]

Jan-Willem. Dutch, like Everts name, don't even try to pronounce it ;-)

Original comment by jwsp...@gmail.com on 24 Jul 2013 at 9:33

[GoogleCodeExporter]

I wanted to express a deep thank-you for implementing this binding.  I have 2 
dimmers, three in-line switches, and one outlet running on this binding and 
they are working perfectly so far.

Original comment by nicholas...@gmail.com on 25 Jul 2013 at 3:28

[GoogleCodeExporter]

OK, I started working on a couple of things in the binding:

- Separation of Command and device classes. They were intertwined in the 
current binding, and are two separate things. The command classes will carry 
the actual implementation of the node commands. The device class is just an 
indication of basic generic and specific device class a node belongs to.

- Error handling. Both on serial level and Control level. On serial level it's 
just resend of last message on CAN or ACK timeout. for controller level it's 
resend of messages up to 3 times and then an exception for synchronous function 
calls and a log message for asynchronous messages. Dead node status is set 
accordingly.

- Threading and queueing. Separated send / receive threads for serial interface 
to remove the thread.sleep function. We now wait on the streams / queue.

- Code cleanup.

Hopefully I'll get this done in two or three days to be able to test / commit 
for 1.3.0.

Original comment by jwsp...@gmail.com on 29 Jul 2013 at 2:24

[GoogleCodeExporter]

Sounds really good Jan (ok to shorten to Jan?!). I am more than happy to test 
once you have the code ready.

Original comment by ben.jone...@gmail.com on 29 Jul 2013 at 8:17

[GoogleCodeExporter]

Me too!

Thanks for picking this up. I just havnt had the time lately.

Original comment by bmcro...@gmail.com on 29 Jul 2013 at 9:23

[GoogleCodeExporter]

Taking a poll:

What zwave controller is everyone using?

Original comment by bmcro...@gmail.com on 30 Jul 2013 at 12:28

[GoogleCodeExporter]

Aeon Labs USB Z-Stick

Original comment by ben.jone...@gmail.com on 30 Jul 2013 at 12:39

[GoogleCodeExporter]

The RaZberry-Zwave-Daughterboard for raspberry pi 
(http://razberry.zwave.me/hardware.php)

Original comment by tlan...@gmail.com on 30 Jul 2013 at 4:39

[GoogleCodeExporter]

Vision USB stick Z-wave

Original comment by jwsp...@gmail.com on 30 Jul 2013 at 9:46

[GoogleCodeExporter]

Another Aeotec Z-Stick Series 2 from Aeon Labs here.

Many thanks for all the hard work guys!

Original comment by riturr...@gmail.com on 30 Jul 2013 at 3:47

[GoogleCodeExporter]

OK. I pretty much finished error handling and separated the device class from 
the command class. I can now even run the binding over a tcp to serial 
connection and I even ran it through an SSL tunnel over the internet. It keeps 
working (though the number of CAN frames increases because of the botched 
timing). 

Bad news is that Command Class discovery and properly implementing Command 
Classes takes a bit more time.

Original comment by jwsp...@gmail.com on 1 Aug 2013 at 8:38

[GoogleCodeExporter]

Sounds good Jan - keep us posted.

Original comment by ben.jone...@gmail.com on 1 Aug 2013 at 11:17

[GoogleCodeExporter]

I've pushed my code so you can see where I'm going with this. The first command 
class is implemented, Manufacturer_specific. 

Original comment by jwsp...@gmail.com on 1 Aug 2013 at 8:34

[GoogleCodeExporter]

And i've implemented Basic as well. At this moment the binding is functional 
again, except for enpoint / multichannel support.

Original comment by jwsp...@gmail.com on 1 Aug 2013 at 10:25

[GoogleCodeExporter]

I've ordered a battery sensor to try and implement a wake-up queue.

Original comment by jwsp...@gmail.com on 2 Aug 2013 at 2:47

[GoogleCodeExporter]

Jan - I will wait for you to implement the multi-instance support as I need 
this for my implementation. Once done let me know and I will download and 
build, and resume testing. Sounds like you are making great progress on this! 
What devices do you currently have and are testing? I am thinking of getting a 
door/window sensor (FGK101-107) and maybe a universal sensor (FGBS001). If so I 
could look at helping implement support for these, unless you already have this 
covered?

Original comment by ben.jone...@gmail.com on 4 Aug 2013 at 11:11

[GoogleCodeExporter]

I'm finished with Multi-channel / multi-instance implementation. Testing would 
be great. I have a couple of Fibaro dimmers (FGD211), a double relay switch 
(FGS221), a single relay switch (FGS211) a roller shutter controller (FGR211) 
and a couple of battery operated sensors from everspring.

I'd love to get the following things done this week, so it can be wrapped up 
before 1.3:

- implement SWITCH_BINARY and SWITCH_MULTILEVEL command classes. Right now I 
send BASIC to every device. This should actually be quite straightforward, 
because the command classes are separate now and I think they can both inherit 
from BASIC. Then I can also add INCREASE / DECREASE support to the multilevel 
command class using the start / end levelchange commands.

- implement a wake-up queue and the corresponding node stages / checks on init 
to support battery operated devices. I bought the evergreen sensors for this, 
they work on 3 AA batteries. 

So I was planning on implementing SENSOR_BINARY anyway, because I need it for 
the evergreen sensors, but if you want to help, that would be great. I'm 
planning on taking some vacation next week, and am pushing a bit to get this 
done in the evenings before my vacation. So you'd have to hurry getting these 
sensors, or it might be in there already ;-)

Original comment by jwsp...@gmail.com on 5 Aug 2013 at 7:17

[GoogleCodeExporter]

Sounds great Jan! I have ordered the sensors but in reality I won't get them 
till Wed/Thurs and won't have a chance to look at any coding until the weekend. 
I have ordered a door/window sensor which is battery powered, and can also 
contain a temperature sensor so not sure how that is handled but will find out 
soon enough I guess! Keep me posted with how you get on and I will try my best 
to do some testing and development over the weekend!

Original comment by ben.jone...@gmail.com on 5 Aug 2013 at 10:24

[GoogleCodeExporter]

I think I have pretty much achieved all goals I set out last week. The binding 
is in a usable state again. I will compile new binaries tomorrow, so that we 
can all test again. This is the list with updates since 2 weeks ago:

- dimmer support
- multi-instance support
- multi-channel support
- starting / stopping of the binding using the equinox console
- watchdog timer to restart the binding on a failed serial connection
- log levels adjusted (no more info messages)
- proper CAN / NAK handling
- frame timeouts while sending / receiving messsages
- separated command and device classes
- Added documentation to all the code
- Separated serial thread and event thread.
- Command class discovery and mandatory command classes per device type.
- Multi-instance / multi-channel discovery (how many instances / endpoints)
- Command Class version support (detect command class versions)
- Implementing concept of node stage and advancing the node stage on 
initialization
- Implemented battery operated devices (wake-UP command class and queue support 
and reporting item)
- Implemented resend queue. 3 messages are retried, then node is set to dead. 
- Implemented callback support (messages are numbered with a callback ID, to 
match with conformations).
- Implemented static initialization support (e.g. fetching static values like 
wake-up interval and sensor type during initialization)
- Implemented dimming to last state versus dimming to 100% on ON command. (see 
https://groups.google.com/forum/#!topic/openhab/0q4GQb_Vs7g)
- Implemented Binary and Alarm Sensor support.

I'll try and get some info on the new features on paper as well. 

Brian: Do you still want to merge the code for final submission and create the 
wiki page? I don't mind doing it, so you could also leave it to me. I'll be on 
holiday from Thursday to Tuesday next week, but afterwards I think we will be 
able to make it for final review / merge in 1.3.0.

Original comment by jwsp...@gmail.com on 6 Aug 2013 at 11:25

[GoogleCodeExporter]

Awesome work Jan-Willem!!

Can't wait to try this out. My new devices still haven't arrived but I will
be having a play this weekend.

Original comment by ben.jone...@gmail.com on 6 Aug 2013 at 11:31

[GoogleCodeExporter]

Simply amazing.

I don't see a need to merge...do you?

I can tackle the wiki this weekend.

Original comment by bmcro...@gmail.com on 7 Aug 2013 at 12:31

[GoogleCodeExporter]

Hmmm... I'm a afraid I was a little optimistic. I ran the binding for a couple 
of hours and I'm losing messages :-(. It seems that i'm overflowing the stick 
with messages and this way some requests get dropped or I get a simple response 
without any indication which message or response it was.

Now the documentation says that you should serialize all messages and wait for 
conformation on everything. I also tried that but this way a sending a message 
takes quite a while: send request, wait for ACK, wait for another request with 
conformation status (callback id), get response message, send ack. It takes 
even so much time that after 2 complete cycles like this my battery operated 
device goes to sleep again. 

Initialization then takes 3 to 5 sleep intervals... that is 3 to 5 hours for 
this sensor. Not really acceptable.

I'll have to think about this. But it will be next week.

Original comment by jwsp...@gmail.com on 7 Aug 2013 at 7:29

[GoogleCodeExporter]

When you say 'I'm losing messages' you mean any messages to any devices, or you 
speak about battery sleeping devices only?

Original comment by belovic...@gmail.com on 7 Aug 2013 at 7:45