Open Fishwaldo opened 5 years ago
Thanks for your works 👍 , i am looking forward to this feature.
- Figure out how Sequencing of multiencapsulated messages is handled with the TransportCommandClass Specs.
Pointers to spec:
SDS13783 Z-Wave Transport-Encapsulation Command Class Specification 2.3.5 Encapsulation order overview Text + especially "Figure 1, Encapsulation overview"
And "3.9 Transport Service Command Class, version 2"
I've looked at quite a few Zniffer captures and I haven't found an example of that CC, but I have been thinking this might get used when doing firmware update (which is not in OZW).
Hi guys, is there any progress/schedule?
I'm interested in S2 and I might help you somehow even though I don't have any idea about how ozw works internally. Is there any developer documentation about how it is structured?
@xeanhort, thank you for your kind offer. I started working on OZW a few months ago so I can relate to your question: "how does OZW (code) work".
@Fishwaldo is the maintainer of this project and I am only helping, but we've discussed a few things so I think I am allowed to chime in...
IMHO if you want to help, this S2 issue is not a good way to start. As you might have noticed in the first post, it is not only a matter of implementing new code, it is about refactoring parts.
I do not want to underestimate you at all, I have no clue about what you already know about Z-Wave. I am going to assume you have basic understanding only.
To be able to understand how encapsulation works, and what the various parts do, you'll have to understand SDS13783, "Z-Wave Transport-Encapsulation Command Class Specification". It is part part of:
The complication is that what begins as a command ("turn on the light") can turn into a Russian doll of packets, and state has to be tracked (so imagine state machines within state machines) which (again, imho, and not underestimating you at all) make it rather hard to debug and follow.
At the moment, there are more open issues, some tagged intermediate or basic... Maybe have a look at those first?
@xeanhort It is quite an involved task... for adding a device in the network there are 30 steps in the documentation, get only one wrong and it does not work.
I managed to implement it for a closed source project, I think it took me something like two weeks full time work for having it working, some other couple of weeks at least to iron things out, not counting implementation for things like transport and supervision classes. Now, this does not mean that somebody else working on such an implementation would have it working in the same time frame, it depends on many things....
Here I wrote some details on what one have to look into for the implementation: https://github.com/robertmuth/PyZwaver/issues/2 It's only public info, you can reach it if you look into the first referred pdf, but that info might save you a couple of hours of digging through that pdf.
I cannot give any details about the closed source project, so...
As far as I can tell there is no open source project that implements S2. I would be glad if somebody could show me otherwise, I would be interested in looking over the code.
@xeanhort It is quite an involved task... for adding a device in the network there are 30 steps in the documentation, get only one wrong and it does not work.
I managed to implement it for a closed source project, I think it took me something like two weeks full time work for having it working, some other couple of weeks at least to iron things out, not counting implementation for things like transport and supervision classes. Now, this does not mean that somebody else working on such an implementation would have it working in the same time frame, it depends on many things....
Here I wrote some details on what one have to look into for the implementation: robertmuth/PyZwaver#2 It's only public info, you can reach it if you look into the first referred pdf, but that info might save you a couple of hours of digging through that pdf.
I cannot give any details about the closed source project, so...
As far as I can tell there is no open source project that implements S2. I would be glad if somebody could show me otherwise, I would be interested in looking over the code.
@aromanro If you would publish some test vectors, like the ones you asked for in https://github.com/robertmuth/PyZwaver/issues/2#issuecomment-446893374 you could accelerate the implementation of an open source project that implements S2, which in turn might benefit your closed source project.
The test vectors I mentioned are available in referred RFCs/papers.
Did you find it easy to match up those papers and RFCs with the S2 description? Was it easy to extract the relevant information/algorithm from them?
I am just saying, a little editorial work on your side (or Zwave's for that matter) would go a long way and ultimately help you.
Having said that, the real craziness is that Zwave does not provide a reference implementation.
My side signed an NDA, I'm sorry about that. Easy? As easy as for anybody else, I think, those papers are referred in the zwave paper. You just have to follow the links, read them... the hard part was not actually getting those, it's way more than that.
The specifications for S2 are very comprehensive and complete, so that’s not the issue. For me, it’s a time issue (or lack of) and the fact that there isn’t a compelling event to support S2 when we already do S0. It’s a nice to have but not a must have issue.
Having said that, the real craziness is that Zwave does not provide a reference implementation.
The reference implementation for SILabs is the Z/IP gateway. That’s the way they are pushing all the vendors who are doing gateways.
To the point they stopped updating the SerialAPI document (at least from what I have seen) and we are waiting to see if the new 700 Series based USB sticks run the bridge or gateway firmware.
Having said that, the real craziness is that Zwave does not provide a reference implementation.
As @Fishwaldo points out, S2 and "everything else" is in Z/IP SDK and if you want to see the source code, all you have to do is register an account on the Silabs Site, download the Z-Wave SDKs (through "Simplicity Studio", do not download the "Z/IP Gateway Software Development Kit" listed on their site because that is V2.81 and that is an old version) and then you can build your own Z/IP gateway (from source). The issue with that is, the documents are not "open" like the Z-Wave Spec. They cannot be used by OpenZWave without permission.
To the point they stopped updating the SerialAPI document (at least from what I have seen)
It received minor updates only, but it is still valid. I own a UZB 700 series stick.
we are waiting to see if the new 700 Series based USB sticks run the bridge or gateway firmware.
The Silabs stick runs "bridge" only, it got updated recently and there is no gateway firmware, nor is there any Zniffer firmware... No word from other vendors yet afaik. I have experimental gateway support on my OZW fork.
the fact that there isn’t a compelling event to support S2 when we already do S0.
Major reported vulnerabilities don't count? The fact that S0 keys are zero padded and trivially crackable? https://sensepost.com/cms/resources/conferences/2013/bh_zwave/Security%20Evaluation%20of%20Z-Wave_WP.pdf
Given that having S0 is almost exactly the same as having no encryption at all... what exactly would you consider a compelling event?
Nothing in that paper talks about cracking keys, and the padding of keys is only happening during inclusion. (Which is a 1-2 second window when you add a device to your network).
The paper talks about a logic error in the SDK that was fixed eons ago.
Please stop spreading FUD. Once the network key is exchanged with a device, unless you can crack AES, you can’t decrypt anything.
If I’m going to break into your house, I’m more than likely going to throw a brick through your window than wait outside with sniffers hoping you do a secure inclusion.
If S2 is so important to you, instead of criticizing- both OZW and the S2 specifications are open. You could go and implement it yourself and share with the community
That is true. It's vulnerable only when including and you can do the inclusion in low power mode if you want.
Apart from a more secure inclusion, S2 also is a little bit faster than S0 because a nonce isn't needed for each packet and that's about it.
Nothing in that paper talks about cracking keys, and the padding of keys is only happening during inclusion. (Which is a 1-2 second window when you add a device to your network).
There's no FUD here. There are kits available that sit and wait for inclusion. Which happens every time you swap batteries, too. Not everyone lives with 100 meters between them and people around them. Hell, I have 50 feet on either side of my house--nobody else in my neighborhood has that space--and yet I've calculated there's at least 70 people who can leave a device sitting in their home and crack my locks when I replace the battery.
Instead of claiming FUD how about you actually read the paper before you dismiss it?
I find it extremely disconcerting that you're supposed to be developing support for S2, and you haven't even read SiLab's own admission of the vulnerability.
There's no FUD here.
"Fear, uncertainty, and doubt (often shortened to FUD) is a disinformation strategy used in sales, marketing, public relations, politics, cults, and propaganda." - (1)
Given that having S0 is almost exactly the same as having no encryption at all...
That sounds like FUD now... but lets continue:
There are kits available that sit and wait for inclusion.
I assume you are referring to SmartStart... a feature of devices that are not included in a network.
Which happens every time you swap batteries, too.
And now I am certain you are not speaking with any level of understanding of the Z-Wave Specifications or the SDK or how devices are included/working. You obviously are drinking the media cool-aid. So in a attempt to educate, lets continue regardless to debunk these assertions of yours:
When a device is included, it writes the Network/Node etc information to Non-Volatile Storage (nvram or eeproms) for the purpose of not losing that information when batteries die or power is cut. This is not even code that the individual vendors write, but included in the Software Development Kits that all vendors develop their firmware upon. This is even part of the certification, to ensure that when a device is restarted from a power failure, or battery swap, it retains its configuration. If users had to reinclude devices everytime they swapped batteries, or had a power failure, well I dare say that Z-Wave would be a total and utter failure
You must have a bad batch of devices if you see that behavior.. I suggest you RMA devices and get some that work according to the specifications and make sure they are certified.
and yet I've calculated there's at least 70 people who can leave a device sitting in their home and crack my locks when I replace the battery.
70 people that can hit the "Exclude and Include" button on your Lock? Because thats the ONLY way to put a new network key on a device, unless you have some really crappy devices. The "issue" you keep quoting above, is related to a single vendor with a bug in their firmware.
But say they can do it remotely because a vendor fucked up his firmware - Wouldn't you notice it when you attempt to verify that the lock is back on your network and you test to make sure its communicating correctly by locking/unlocking the door remotely?
Instead of claiming FUD how about you actually read the paper before you dismiss it?
Quote from the paper (conclusion section)
Please - point me to the line/paragraph where it states that the S0 Specifications is broken and how they exploited S0 and not some random vendors buggy implementation? Then I'll accept your FUD.
But if you don't believe the author of the paper, (or obviously myself) then how about this:
So, your basing the whole S0 is "Insecure" because a single vendor fucked up in 2013? (not Sigma or the Protocol).
I am taking this extremely personally because: 1) I work on OZW in my spare time. I don't get paid for any of it, and I do it out of enjoyment. Just the simple fact of the matter that there is a S2 tasks here says I will do it, when I have the inclination and time to do it. Personally attacking my lack of time to implement S2 is not cool at all and as I have said, all devices that Support S2 also support S0, so there is no critical event (I brought this device and it doesn't work type event) because of Lack of S2 Support 2) If you are so gung ho on increasing the Security of your Z-Wave Network, patches welcome. 3) As I said above, you are spreading misinformation about S0 Security, and are no better than some random sensationalist journalist that spreads this information.
Now I'm sure you will come back with further examples, so lets just cover the other "security flaw" in Z-Wave (that actually affects S2!)
The 2018 "Downgrade Attack" against devices that support S2. They can be forced into only including as S0 encryption. I (and the Z-Wave Alliance/Silabs) would actually call this "backwards compatibility". (and in fact, because of this "vulnerability" OZW is able to continue to work with these devices at S0 level).
Now once again, the window to exploit this is a matter of seconds during the initial inclusion, and as far as I'm concerned, the vulnerability is really not "mandating" the Gateway Software raises a prompt saying "Tried S2, but got S0, are you sure?" - (and which is subsequently mandated in the specifications.
you haven't even read SiLab's own admission of the vulnerability.
Hahah? What? Admission? Where? Please point it to me. In this case, they acknowledged a vendor fucked up and increased the testing to make sure other vendors don't fuck it. It wasn't Sigma/SiLabs fault! it was some random developer working for a Lock Company who write a few bad lines of code!
For everyone else following this thread - Note - No where do I say S2 is not a big improvement over S0. Reduced traffic, isolation, improved key exchange etc. It will solve challenges users experience with S0 etc. But when given the fact that someone lands a device on my desk that doesn't work at all with OZW, versus a device that does, but at S0 level, guess where my attention goes?
In Conclusion - these types of conversations are the triggers that usually end up with me going on extended hiatus from working on OZW. I do not want money, devices, but people that will make positive contributions (ie, patches, testing etc) to OZW. Being abusive or judgemental against me and my commitment to my job, family and other hobbies I partake in... not tolerated around here I'm sorry.
Unfortunately, your contribution to OZW (and this discussion) is not constructive, is false and misleading and is making demands of me without even the slightest hint of help. For that reason, I will no longer entertain you in this community. Please go find another Open Source Z-Wave implementation to complain to about lack of S2 support.
For transparency sake - I also banned Electrofloat.
This is a bug tracker - Lets limit the discussion to technical bugs and implementation activities. Any converstation/comment that does not have a positive impact upon implementing S2 will also end up banned.
As you can see - I'm at the edge of my tolerance level for ungrateful users making demands, and its these conversations that make me want to quit.
I was just reading this as I was interested in S2 progress. I just want to say don't quit @Fishwaldo . Just because there are some keyboard warriors out there that want to make noise, doesn't mean your work is not appreciated by 1000 times more users. I wish I had the coding skills to help, but all I can do is say I appreciate your work a lot.
Thanks.
Just wanted to chime in and say that I'm very eager for S2 support. Not just just for the objectively better security (the practical vulnerability of S0 can be debated) but also the fact that S2 is a lot more efficient. You're effectively replacing the three steps S0 process with a single command structure. My Z-Wave network through Home Assistant is pretty much unusable if I have more than a few devices in secure mode.
I've started playing around with the code base and perhaps I'll motivate myself to contribute.
Lastly, thanks for your efforts developing this!
I would like to contribute to the development of S2, so my 2 cents about that issue, to clarify current state:
firstly, although I won't disagree @Fishwaldo, but...
there are indeed well known theoretical and practical scenarios where S0 communication (also after pairing process) can affect the stability of whole z-wave mesh (for example attacker flood with "nonce-get" frames causing a "blockade" of legitimate packets, so nothing works anymore, or for instance a "blackhole" similar routing attacks against the gateway or repeater nodes, etc); the only way to prevent it, is to disable S0 completely.
besides obvious pros of S2, it'd also simplify to switch to HASS, OpenHAB or other OSS projects from proprietary gateway software or other products/projects (used S2-based pairing initially) just using import of their network-keys (S2Authenticated, S2Access), without to rebuild the whole network (just imagine an effort of reassignment from scratch of some large network with 50 or even 100 devices).
hasn't someone already started some experiments to implement S2, so could probably navigate me to some (experimental) branch or a test attempts, I could review or continue?
@PoltoS, is there some way to port it from your z-wave.me project (at least in parts) or to get some help or tips (as I saw somewhere once you wanted to provide an integration to HASS and did fork openzwave repo too)?
@Fishwaldo et al, isn't there any way to open resp. to encapsulate the security layer (or responsible communication part) in the interface of OZW, in order to relocate that to outside, e. g. to provide first tests or even implementation in high-level langs (like python), so some binding (like python-openzwave) may obtain it earlier and development/experimenting may become much simpler due to superfluity of build and deployment?
@sebres
TLDR: https://github.com/Z-Wave-Me/ozway/tree/debug
We at Z-Wave.Me are indeed working on an integration in HAss/Domoticz/..., but it is not done by improving OZW, but rather by doing a wrapper around our Z-Way library that looks like OpenZWave from outside. We call it OZWay ;) Just a simple replacement of libopenzwave.so does the job to switch to Z-Way library.
We are not too close to the end yet, but many things are already working. The main idea of this work is to simplify the transition from OZW to a certified Z-Way that supports all the modern Z-Wave features including S0, S2, Smart Start, Multicast, OTA IMA diagnostics and many more.
This definitelly contradicts the idea of OZW - we substitute OSS by proprietary non OSS code, so I expect a lot of hate ;)
Also we will still require people to use Z-Wave.Me hardware since we base on own firmwares in the ZGM130S or ZM5101. Those two points (non OSS and our h/w) might turn people agains it. Would be interesting to know the community opinion about this.
Anyway people do need modern Z-Wave features and we know how to fullfill them. There are many reasons to have S2 and other nice features present in Z-Way and this is not only about security but also about diagnostics, OTA, performance, ... And after all, paying 60 Euros for the controller h/w when you spent 1000-2000 for peripherals makes sense if you get more out of those peripherals.
Don't want to run in this discussion here (Justin will reasonably ban me for this ;) So if somebody wish to open this discussion, we can move it in private mails or our forum.
Our roadmap is to fork the OZW code and rewrite it step by step throwing away all Z-Wave specific code from OZW keeping only OZW Notifications and public API calls. So far it is almost working with few CCs in HAss and once fully working with those CCs, we will share it with the community to speed up the finalization on other CCs (will be a lot of cope&paste work for each CC with OZW specific Notifications and ValueIDs).
If there are people ready to help us, we would be glad. The most wanted thing for now is to explain basic concepts of OZW and understand what is public and what is private/internal. Because OZW do not really provide good separation of public vs private calls (yes there is Internal namespace and many private methods/properties, but still a lot public calls should never be exposed as public and we don't really know why are they made public - this knowledge is important for us to work only on real public API and remove all internal code). Same for the concept of XMLs - what are XML products descriptions used for, what is stored in the file storage beside the interview results (that are stored in the underlying Z-Way).
Another good thing might be to get prooven test projects based on OZW and try them with OZWay.
Our next step is to even provide a way to run this wrapper and Z-Way main code simultaneously (use RPC between two instances of Z-Way lib: one in Z-Way, one in OZWay). This will allow to use the power of Z-Way diagnostic tools and Z-Wave Expert UI keeping the HAss/Domoticz/ozw-console/... UI simple and not to require the integration of all Z-Wave specific UIs for Smart Start, S2 and other modern Z-Wave features in home automation software. This is the concept we use in our Z-Way and most people salutes it.
If somebody want to have a look on what we currently have, here is it: https://github.com/Z-Wave-Me/ozway/tree/debug. We have forked a recent master and merged our changes in it.
As for helping you in implementation of S2, this is a hard job! We did it in 9 months being really focussed on it and using S2 lib by Sigma (now SiLabs). And then another year to polish it in real world tests. And it required us to rewite parts of Z-Way core (even we had a very modular structure and were prepared for S2!), implement Transport Service (very strange CC in terms of design - does not fit main Z-Wave CC design), Supervision (another strange CC that required changes in ALL CCs), rewriting S2 library by Sigma (fixed few critical bugs, undocumented features, huge design drawbacks and limitations). To make it certifyable you also need the Inclusion CC (another freaky CC).
It is certainly possible to do that with OZW, but from what I've seen, it will require full rework of the Msg class introducing more transparent encapsulation concept (Z-Wave has a lot of encapsulations: Multichannel, MultiCmd, CRC16, S0, S2, Supervision, Transport Services - none of them have good modular implementation in OZW). Worth? Definitelly. But who has enough motivation/time/investments to do that? As I got, Justin is the only maintainer last years and there is no enough motivation to do S2 for now.
@PoltoS did you miss the LGPL license on this project? You can't just take OZW, fork it, rip out entire sections of it replacing it with closed source code, and then sell it.
@jcam No, we did not. We keep our modification entirely under LGPL. We do not close any source code from OZW.
To clarify, our concept is another: we modify the LGPL code of OZW so that instead of doing direct Serial API calls we call another library (our Z-Way library). So this does not violate the LGPL license.
Additionally, we do not want to sell OZWay (the modification of OZW relying on Z-Way library). We donate it to the community (this is why we decided not to write from scratch but rather base on exisiting OZW code).
Please correct me if I'm wrong, but for me it looks pretty clean.
I guess clean is relative... Taking advantage of a community free software project and leveraging it for the purposes of bringing more sales to your zway library licenses doesn't feel very 'clean' to me, but I guess this was quicker than just writing a full MQTT interface for zway <-> homeassistant. The "donation to the community" you mention is only of use if someone buys your library... or are you planning to release zway for free and let the community use it with other zwave devices? That would be a great donation!
This discussion is far beyond the topic of the issue. Is there a better place for it?
@jcam, let's separate legal side from your feelings ;) Legally we are clean.
Many of OZW users do use RaZberry and they will not have to pay us any extra penny. So for them it is a good option.
Trust me, we do it not for end users. Many end users will certainly use it, but we are focused on companies who were trapped by the OZW in the past. They have huge problems with certification now (unlike end users, they must pass certification) and are confined by OZW restrictions in regards to S2 and other important Z-Wave features. And there is no paid support for them from the OZW community as it just can't deliver a certifiable code. So we are taking a niche OZW community is not interested in for years. We are not evil, we just want to help existing companies to start doing better.
Many companies find OZW as a good start, but it is not once they hit the market and need to deliver a stable certifiable code.
IMHO, the problem with OZW is the absence of paid support. And those who use it in their projects do not contribute back to the master branch - they keep it for their own (even if released publicly according to LGPL, nobody do really merge their work in master branch). So the project is relying on few enthusiasts with no money motivation. And after few years of such situations the project is many years behind the modern Z-Wave. And not sure this is easy to solve without introducing business customers.
I think perhaps you're right that this is not the right place for this discussion. This thread is about bringing S2 support into OZW, the "free software library that interfaces with selected Z-Wave PC controllers".
Your project does not bring S2 to OZW, it replaces OpenZWave with your paid closed commercial product, using bits of OZW as a wrapper API. It has a definite use, especially since there are not enough contributors to the original project.
If there are people interested in doing S2, I can for sure help with advices, some guidance and answers to specific questions to one-two OZW maintainers, but I have pretty limited understanding of OZW messaging internals and the overall work plan and codeing should come from OZW maintainers. I don't believe the community can do such a huge work thar requires very deep synchronized code changes - one-two, maximum three people can do it. One programmer have to alter so many parts in the code, that having multiple programmers will lead to merge conflicts and unneeded complexity. It is not a bazaar project IMHO.
We on our side will continue our work, which make sense for many people and companies. And I would appreciate help from OZW gurus.
@jcam I am a user of ZWave and need functional system. I have been following OZW project since I started home automation in 2016 and tried to contribute from my say "advanced user" level - as I am not a programmer. I would stayed happy with OZW if it was evolving but it is not, @Fishwaldo is not here for very long periods. Pull requests and Issues solving is depending on the manager, out of office, so to say. Personally I prefer to "risk" $ 70 in Z-Way and get the chance to take better control over my $ 2500 Domoticz ZWave system. I am convinced @Fishwaldo have the best intentions with OZW but I have no time to wait any more, I started to buy ZWave hardware because of my believes in OZW plugin to Domoticz but quite early I had the fear OZW was not the project it was marketed.
@PoltoS Thx, I'll take a look at your debug branch and all the ZWay stuff. I guess also that it is not trivial, but hope to find some way to simplify the things (in order to encapsulate it in parts or to relocate it to python and co).
@JGR-66 et al ("not a programmer", with intention to flood here with such non-constructive comments) - please remain factual. Neither here is a place for a frustration claims, nor the cost of your system do matter at all... This is just contra productive. Also try to be nice to your developers, doing that for nothing, frequently tired after work, in a leisure time unable to enjoy this time together with their families.
@sebres We have changed it to https://github.com/Z-Wave-Me/ozway/tree/OZWay branch to make it more clear. Let's move our discussion here: https://github.com/Z-Wave-Me/ozway/discussions/2
@sebres I appreciate developers a lot! And I make donations when I find other unpaid peoples hard work useful for me. Anyway there must be some room for discussions how a project is evolving. Especially OZW as so many people are depending on it, Open Source or not. Otherwise the project website should be very careful in marketing saying very much, what to expect from the software... I was banned here before just because I forwarded some criticism, if you are in control but do not have the time etc, why not let other people in to help instead! Of course this thread is really not the right place, but I believe if a new thread with correct topic was started it would be deleted and everyone in it banned ;) Have a nice day!
here must be some room for discussions how a project is evolving
by the way (@Fishwaldo or repo admins/owners), github got a new feature called "Discussions", so may be it could be enabled for this and other repos here, so people can discuss there some concerns (without to bother development or overfill the issue tracker).
I was banned here before just because I forwarded some criticism
I don't know the whole history here or what was happening exactly. Just issues are definitely not a "threads" in sense you may understand it (anyway not a discussions or whatever usually taking place in forums or mailing lists). Holding an issue clear and factual is very important (at least spares a lot of time coming back to some issue).
I'm not sure if this is an option, but the latest Aeotec Z-Stick Gen5+ claims to do S2 "natively." I really don't know what that means, but given you've always been able to do S2 in software with the previous Gen5 it must mean something! Clearly it'll need some sort of different API call or parameter at the very least, but it might be a much simpler option than having to implement it fully in OZW?
Also, and this might be a stupid thought since I know next to nothing about the workings of the Z Wave API, would a node already included using S2 and in the Z Stick's configuration appear the same to OZW as anything else? If so, there would be the possibility of including the device directly on the stick with no support from OZW at all?
Task List for S2 Support. Lots of Refactoring is going to need to happen before we can start implementing it....