Human readable types vs machine optimzied types

[oliv3r]

Homie pretends to favor human readable forms, by making everything a string.

However, MQTT is in theory a M2M protocol, including very simple and basic sensors and actuators (microcontrolers).

Sadly, we are stuck now in the middle and the definition of the protocol is awkwardly defined.

Integers are to be represented in strings, which is nice for human readability, but is more expensive in bandwidth (it takes more bytes to encode the strings) and also more expensive to encode/decode (atoi/sprintf). However, this is of course a choice, to favor human consumption.

On the other hand however, and this is especially true for the boolean type, the spec only allows one form, where it is relatively easy to allow t, f, T, F, 0, 1, <positive int>, true, false, TRUE, FALSE. Yes, some additional code is needed, but it is more human friendly. Especially the first-letter-only approach at least saves some additional bytes.

So what is the goal here? Is human consumption preferred, or should the focus be more towards M2M, where battery life, cheap hardware is potentially more important?

What is the design goal of the homie convention?

Note, that overall the convention is very sensible ;) and I think the IOT world most definitely needs something like this.

[krocans]

@oliv3r regarding boolean type I think you are proposing "improvements" which will have absolutely opposite effect. As I've commented in issue #213 if we add something to convention each an every library nut have support for new data types/values. If we count all protocol overheads there will be almost no (1% or less) bandwidth "economy" from replacing "true" with "t" or 1. And it gives no performance gain or something. Form other side implementing support of all those t, f, 0, 1 in libraries as you correctly said needs some additional code. And there is a problem - with additional code we are wasting resource which in opposite to bandwidth really is important and limited on cheap hardware - memory. One one day lib supporting all that bunch of formats and value representations will simply not fit in your cheap hardware.

Also homie layer does not care about meaning of data being transmitted. It doesn't care what is that 23, "true" or something else. An that makes it flexibe enough. You can simply make number property and transmit 0 an 1 there and let your app understand it as boolean in applications logic. Or send string type "t" and "f" an use it as boolean. I am in some cases using it this way and it's ok for me.

But your proposal just gives me more idea. Probably it's worth to include in homie some data type like "binary" or "void" which will mean absolutely unstructured data (from homie point of view). So Homie does not define any values or validation for this type only particular app cares about it. That will allow for everybody who needs some new/specific data formats implement those in app remaining 100% convention compatible. And there will be no need for dirty hacks like use string as boolean and no need to ask an discuss including in convention each and every fancy datatype needed maybe by one person in world.

[oliv3r]

Hey @krocans not improvements yet, but I want to clarify a few things and get a clear statement on design decisions.

1% or less in terms of bandwidth is of course hugely depending on the scenario; but it's generally speaking more then that, depending on how well it compresses due to encryption. /path/value=true we are already talking 10% overhead. Regardless.

You speak rightfully so, memory, flash and bandwidth being expensive, but even CPU could be considered expensive. But remember, waking up from sleep, turning on a modem/WiFi, send a few bytes and go back to sleep. Here bandwidth, every byte transmitted, is very costly on your battery. Yes memory and flash are even costly here.

So what I was after, was not so much as to fix anything (yet :p) but first understand what the design goal is. Is it (like mqtt is) a M2M optimized protocol? e.g. why not send 0/1 or just t/f? If it is with human readability in mind, why not allow a broader range. Just allowing 'true/false' sits exactly in the middle. It's human friendly, but just about, it's not super machine friendly.

Also 'accepting 't' and 'f' means this could be handled in a backwards compatible manor, as when 'true' is sent, you can still have a valid match on the first character (which is still much cheaper to process then strncmp) which is still also cheaper on memory/flash. So in my opinion that would be the middle-ground. It's less efficient then sending '0/!0' in binary, but you'd only have to transmit a single byte for your value (t/f) which is still human readable.

Your point with regards to 'void' data does indeed make sense too 'arbitrary binary data', which neither to homie nor to the relay matters, but the receiving end may no exactly what to do with it.

The downside however to your approach is, to 'abuse' it to send random data, which then very easily (too easily?) opens the door to just deal with everything in your app, and then it becomes one of those standards, that nobody addhere's to, as everything is in a 'custom vendor message' ... (not that you couldn't right now, just base64 encode your data :p but that makes your processing/battery life very expensive)

Without a clear design goal, the above makes little sense, and improvements are almost impossible to do, as we don't know which direction o engineer.

[ajxn]

Well, I do think the overhead of a UDP package sent is way more then the difference between '1', 't' and 'true'. So the encoding is not that huge part of the overhead when it comes to the IP protocol. You might have a point when it coms to handling the data, but not when it comes to transfer data with IP or Bluetooth or similar protocols. There the time making a connection and then transfer and handshake is way more expensive. So if you have the connection up, it doesn't matter that much if you send 1 or 100 bytes, as the package for TCP/IP has a minimal size of packages sent.

Unless you have a compact serial line p2p protocol.

Calculation to prepare data and interpret data is more important when it come with CPU usage, and so is memory use.

[toolsfactory]

Not sure if development of the protocol is still active. It would be a pity of not as it bridges quite well a gap in many home automation systems. Regarding this topic, i have a similar problem and also a potential way forward that could also @oliv3r 's request. In my case, i have quite some data types that do not match any of the ones provided by homie. For example one type defines a time within a week. USed for things like every tuesday 3pm do something. A nice way to work around this is if support for the official KNX Datapoint types would be added as they cover pretty much every edge case and the data is encoded in bytes. The solution could therefore be to introduce an new datatype "dpt" where the format represents the dpt id like "9.001". dpt encoding and decoding libs are available for nearly every device in highly optimized versions and all home automation systems know how to handle them. so supporting this should be relatively straigt forward.

[oliv3r]

Ahh some necroing :) Anyway, I think my most important point was; to be consistent. Either do it one way or the other; but don't mix and match. The talk derailed a lot in talking about 'oh it only costs so many extra bytes' which are all valid discussions. But if we are talking about 1000 messages a day, we are talking 1000 vs 100000 bytes, which also includes energy usage.

Anyway, consistency was the keyword :) So if human readability is preferred, allow all formats, and deal with it in software. If you want to be 'cheap' (bw etc) don't make everything a string :)

[Thalhammer]

Imma drop my two cents here since it seems like I am the only one of the maintainers left still having notifications on. In my opinion homie should have done away with the focus on having tons of topics, which are hard to keep in sync and adopted some sort of structured coding inside a single topic a long time ago.

For example things that represent hard stuff of the firmware (.e.g implemented version, name, extensions, implementation...) will never change individually and should instead be a single topic. Similar reasoning applies to the descriptions of properties and nodes. The datatype of a property shouldn't really change at runtime and if it did, most implementations probably wont handle it correctly anyway.

Personally I would have favoured protobuf, since its trivial to parse and there are libs for all device classes available. More important than being small/fast though is that protobuf by default is both backwards and forwards compatible, meaning you can iterate on the definitions without breaking existing stuff.

However it never came to that. Changing it now would result in what is essentially a new protocol and with the plethora of modern iot protocols created in recent years I am not sure there's a need for that.

If you want to use homie but need binary data you could base64 encode it.

[Tieske]

@Thalhammer would the team be open to add new members to take this project forward?

(Apologies for hijacking this topic)

[Thalhammer]

(Apologies for hijacking this topic)

No problem

@Thalhammer would the team be open to add new members to take this project forward?

Sorry for the late reply. The "Team", which currently consist of davidgraeff, lorenwest, me and marvinroger (the original author of the project), seems basically dead. I wrote an extensive post in the team discussions about how I would like to move forward and the only one that replied at all was davidgraeff and he pointed out that fixing it would impact all implementations that need to enumerate devices. On top of that AFAIK none of the other members seem to still use (or care) about homie and to be perfectly honest I'll probably move to a different (probably custom) solution as well once I have the time to do so.

The proper way to fix it (as I have outlined a number of times) would probably result in what would be a more or less new protocol. I am not sure what you had in mind with "take this project forward" (i.e. fix the issues, add minor features or fix it properly), would be cool if you could elaborate a bit what you thought about.

I would absolutely be available for a homie 5.0 that keeps the good ideas and fixes the bad parts along the way, but I am not sure how doable it is given that some of the issues are deeply rooted inside the underlying mqtt protocol and unlikely to ever change.

[jamesmyatt]

Probably worth converting this diversion into a discussion. But in the meantime...

As a former Homie user, I feel like you need to justify why people might want to use Homie vs. any of its competitors, especially ESPHome, before you invest any more time and effort in developing it. ESPHome is far more active, more popular and better funded, and, in most cases, it's easier to use and more powerful. For example, I see 2 videos on YouTube about Homie vs hundreds about ESPHome.

[ThomDietrich]

Hey all, I was also quite involved with the project some years ago. Back then we had high goals for it and set everything up so wider adoption was a viable option.

Re your comment regarding ESPHome @jamesmyatt: To be fair, Homie was never intended to be in any competition with ESPHome or any of the other famous ESP and IoT frameworks out there. Homie is a convention for MQTT communication and as such should have found it's way into frameworks like ESPHome or controllers like Home Assistant. The fact that it didn't made it rather irrelevant for any other small project which could have benefited from its maturity and promise of frictionless integration.

I still believe Homie was an amazing idea and a great exercise to be involved in. In reality my newer projects rely on the Home Assistant MQTT discovery "convention", to conveniently integrate with it natively.

[jamesmyatt]

@ThomDietrich exactly. The Home Assistant discovery protocol is far better supported. And esphome provides the boilerplate for 90%+ of what you want to do, which would be in the homie implementations. So the question is what's left?

[Tieske]

@jamesmyatt ESPHome has a narrow focus on ESP type hardware. Which is completely different from Homie, which is aimed to be a generic convention independent of hardware/software platforms. It only ties itself to MQTT.

As for Homie future; I wrote my thoughts on this in https://github.com/homieiot/convention/issues/228 so let's stop hijacking this issue

[jamesmyatt]

@Tieske You're quite right. I suppose I still see Homie as primarily the original ESP framework, with the convention as just documentation of the MQTT interface that it implements.

[Tieske]

Since there is a proposal now for Homie 5, based on a single topic description based on JSON, and design goals specified in the readme. I think we can close this issue now.

Please reopen if deemed necessary.