Electric board channels: multiple values in an array

xen2 commented 1 year ago

Great work on the project!

I was working on adding my Panasonic "Smart Cosmo" electric distribution board to Home Assistant.

The thing is, one single ECHONET value returns up to 60 values in an array.

On the pychonet side, I wrapped the 60 values in a JSON array.

On the home assistant plugin side, I did the following "hack": https://github.com/scottyphillips/echonetlite_homeassistant/commit/56c3c93993d4143caa524dffe584be7152430fdd#diff-e82565bd3ee78c190e10f8e620df9a97630dfb81dc19fd88f2aeefedc785586dR86

Basically, for a single ECHONET value, I allow the creation of multiple sensors with a "subpath" (resolved by a lambda) on the ECHONET property. Does it seem like a good approach, or do you have any other idea? Note: if we stick with this lambda approach, it could be used in various other cases to simplify a bunch of if/else for special cases, unit conversion, etc.

Anyway, using this approach everything is working, I can see the consumption of each of my individual 40 lines in watts.

Happy to do a PR once everything is working and clean!

scottyphillips commented 1 year ago

This looks great - Please put in a PR when ready! There is a number of echonetlite EPCs in the wild that aggregate data such a manner. @nao-pon did you want to take a look also.

I have stepped away a little from maintaining this at the moment but I have no dramas with how you have approached this.

nao-pon commented 1 year ago

@xen2 Thank you for your contribution. it looks good. One question is why did you include the lambda value instead of just the index. Is there anything wrong with getting by index from a dictionary object?

Also, I would like to know the difference of pychonet.

Thanks! 👍

nao-pon commented 1 year ago

@xen2 I am very interested in your implementation. I've made a few tweaks and created the following branches: (I also include support for the LowVoltageSmartElectricEnergyMeter class for testing purposes)

https://github.com/nao-pon/echonetlite_homeassistant/tree/epc_multi (DIFF)

        0x87 : {
            0xB7: {
                CONF_ICON: "mdi:flash",
                CONF_TYPE: DEVICE_CLASS_POWER,
                CONF_STATE_CLASS: SensorStateClass.MEASUREMENT,
                TYPE_DATA_DICT: 60
            },

Specify TYPE_DATA_DICT: 60 in const.py and return JSON dictionary with number as key and state value from pychonet.

xen2 commented 1 year ago

Thanks for the follow-up! The reason I used a lambda is:

For this specific case, I needed to access value['values'][index] but the JSON message could have different names/structures
It could also be useful for type conversion (i.e. some kW to W) in the future. For example, some of the special cases hardcoded in EchonetSensor.native_value could be directly handled with this system.

But I am fine to do it differently.

I have published the pychonet branch: https://github.com/xen2/pychonet/tree/distribpanel_individual_channels

I will be doing a bunch of other patches soon, I have a bunch of other devices to connect (enefarm, solar panel, batteries, etc.).

nao-pon commented 1 year ago

I checked the specs and it seems that out of the number of available channels (252), the maximum is 60 channels starting from a specific channel number. Need information on this channel number?

In my implementation, the starting channel number and the number appended to the entity name displayed in the HA may not match.

Regarding the specification of the unit, the value specified by CONF_UNIT_OF_MEASUREMENT has high priority.

https://github.com/scottyphillips/echonetlite_homeassistant/commit/9ef2ad1aeeaffdfb294ee912439a6148a0ee60a2

xen2 commented 1 year ago

In my case I needed only 40 so I didn't bother to handle "pagination".

Otherwise, what needs to be done is splitting in multiple requests: Query number of channels with 0xB1

Then start a loop (assuming 100 values, it would do ranges 0-39, 40-79, 80-99)

set desired range with 0xB6
then query values with 0xB7

Even if we don't implement all of that, it would be good (but not necessary yet) to at least do + value.start (which should be 0 only so far) so that it doesn't break if we add pagination later.

xen2 commented 1 year ago

English manual for reference:

Sources: EN: https://echonet.jp/wp/wp-content/uploads/pdf/General/Standard/Release/Release_Q/Appendix_Release_Q_E.pdf JP: https://echonet.jp/wp/wp-content/uploads/pdf/General/Standard/Release/Release_Q/Appendix_Release_Q.pdf

nao-pon commented 1 year ago

Well, it's a little complicated. Does the value of EPC:0xB1 match the actual number of circuit breakers you are using? If so, creating that number of entities at startup eliminates wasted entities.

In your const.py configuration you need at least the values EPC:0xB1 and EPC:0xB6 to get this EPC value. config requires a definition of how to use them, and we have to think of a way to make it a general-purpose config.

Also, when the number of distribution boards exceeds 60, it is a very special case. For the time being, I think it should be left as an issue for the future.

xen2 commented 1 year ago

Does the value of EPC:0xB1 match the actual number of circuit breakers you are using? If so, creating that number of entities at startup eliminates wasted entities.

Yes. Or even easier, it's part of the result of 0xB7: { start: 0, range: 40, values: { .... 40 values here ... } }

Assuming you don't plan to support values past 60, you can use the range: 40 result part of 0xB7 answer. That's why I did index: value['values'][index] if index < value['range'] else None in my lambda: it returns None if values are out of bound, so even though 60 sensors are created (because I don't know at creation time), the last 20 are disabled. Of course if we could ask 0xB1 and wait answer before sensor are created, that would be even better (no need to create non-existing sensors 41..60).

nao-pon commented 1 year ago

I'm considering a config like this:

        0x87 : {
            0xB7: {
                CONF_TYPE: DEVICE_CLASS_POWER,
                CONF_STATE_CLASS: SensorStateClass.MEASUREMENT,
                TYPE_DATA_DICT: {
                    "getnum": 0xB1,
                    "setdata": {
                        0xB6: 0
                    }
                }
            },

"getnum" is the EPC number to get the maximum value when the returned dictionary is an arbitrary number.

"setdata" will be checked before GET and sent in advance if different.

I'm still thinking about the implementation. Please let me know if you have any suggestions.

xen2 commented 1 year ago

Looks good for the config. Let's see how we can generalize it to other cases later. What does the 0xB6: 0 mean? is this the start index to set? I would name it setrange. In practice I don't need to set it, it returns 0..40 by default in my case. Anyway, we probably can remove the 0xB6 setrange for now (cf next part). 0xB1 should be enough if we don't support past 60.

Ideally, I would say that we would like to run the 0xB1 command during the device discovery phase (async_setup_entry()). It looks like we could use response = await self._api.echonetMessage(...) inside if TYPE_ARRAY_MAX_SIZE in _keys: block of async_setup_entry() (which is async) to achieve that. I don't think we need to call 0xB6 at all, we can just log a warning if 0xB1 is larger than 60, saying we won't be creating/recording sensors past the 60th.

Then we can proceed with the sensor creation loop once we know the exact number.

nao-pon commented 1 year ago

Oh, 0xB6 should also include the range to get. I am concerned that other devices may change the value of 0xB6. You can limit the range to 60. For this library, just throw it away.

It's 0xB6: b'\x00\x3C'.

xen2 commented 1 year ago

Good point, it could be changed behind our back.

I don't think you can set the range to 60 if only 40 channels, you probably have to set a range that is max 0..40, not 0..60

I am not sure if possible, but to avoid another setrange being set behind our back, maybe there is a way to do a request that set the 0xB6 value and read 0xB7 in a single atomic transaction? I took a quick look at the spec and found a SetGet but that's probably only for the same property. Otherwise, simply setting 0xB6 once and reading 0xB7 is probably good enough.

Either way, even if we don't call 0xB6 setrange but only 0xB7, we can still be safe if we interpret both "start" and "range" from 0xB7 and use it when updating values, i.e. lambda could be: value['values'][index - value['start']] if index >= value['start'] && index < value['start'] + value['range'] else None That way, we can at least be sure to not update the wrong index if "start" is not 0 (worst thing that can happen is that we don't update 0..60 anymore, but since we don't have case with 60+ channels yet, this shouldn't happen yet -- we can even issue warnings that it's not supported yet if it happens).

nao-pon commented 1 year ago

@xen2 After much consideration, I think your implementation is currently the simplest.

For the time being, I would like to support such properties as long as they are not paginated, and if there are any more, I would like to be undefined.

I look forward to your PR. thank you! 👍

xen2 commented 1 year ago

I will try to improve it little bit and submit a PR then! I wouldn't mind implementing proper pagination but I don't have a device that requires it so it's difficult to test.

Also, I looked at Smart Cosmo catalog and nothing goes beyond 43 channels. If you need more than 40, you need to install a second board and I think it would come up as a second device, not merged with main one (I will update you on that soon since it will be my case: I have a 2nd electric board. It doesn't have the ECHONET module yet but it's going to be added soon). As a result, it seems like 60+ will never happen anytime soon.

nao-pon commented 1 year ago

Earlier, I merged the implementation of TYPE_DATA_DICT, so I think that it corresponds to the values of EPC 0xC7, 0xC8 of 0x02-0x87.

Could you please check it as well?

https://github.com/scottyphillips/echonetlite_homeassistant/commit/d6065bbeb68c6a5f201d1bac2c82d2cc1fbc1086 const.py

        0xC7: {
            CONF_TYPE: DEVICE_CLASS_CURRENT,
            CONF_STATE_CLASS: SensorStateClass.MEASUREMENT,
            TYPE_DATA_DICT: ["r_phase_amperes", "t_phase_amperes"]
        },
        0xC8: {
            CONF_TYPE: DEVICE_CLASS_VOLTAGE,
            CONF_STATE_CLASS: SensorStateClass.MEASUREMENT,
            TYPE_DATA_DICT: ["r_sn_voltage", "sn_t_voltage"]
        },

xen2 commented 1 year ago

OK, I will double check if I receive the numbers.

However, based on my experience so far: I doubt me (or anybody else) will need any of those numbers (voltage/amps, especially with phase R/T which I couldn't make sense of, I couldn't find any easy formula on the net, maybe it's only in Japanese or naming is specific to Panasonic? I am not knowledgeable enough in high voltage electricity). Also, before I saw the watts values (0xB7), I used those volts/amps values to figure out watts (P=V*I formula to get watts from voltage + amp) but quickly gave up: values were too inaccurate for usage since it's multiple of 0.1A, so at 100V it's basically rounding every 10W). Most of my home had values between 0 and 0.5A for a typical plug, so it wasn't great.

I would recommend removing those to avoid polluting/confusing users with unnecessary sensors.

In practice, I think most people work with watts to do power consumption measurements and predict electricity bill/usage.

xen2 commented 1 year ago

Great job on the TYPE_DATA_DICT though, it will be useful! I will try to rebase on top of that.

nao-pon commented 1 year ago

I think that the values of various sensors are divided into those who need them and those who do not. It is true that current and voltage may not be necessary in general, but we believe that there are also requests such as wanting to find the power factor from the difference between W and VA, or wanting to see the balance between the r-phase and the t-phase.

For less common sensor entities, we can also disable them as default, so I'll look into that.

EDITED: It's Done in 8c8e412

xen2 commented 1 year ago

Note: I didn't forget about this PR but waiting for the second electric board to be setup to finish it.

nao-pon commented 6 months ago

@xen2 I merged your branch electric_board_channels into master and created a PR with enhancements for the "light", "fan" entities.

You will need to use _my edge branch_ to test this as it also requires an update to pychonet for it to work. Could you please verify using it?

nao-pon commented 6 months ago

Currently, I am adding Time entity, Number entity, and expanding Select entity related to #161. These are likely to conflict with this PR #160, so I'll merge it into master. Please let me know if you have any problems.

xen2 commented 6 months ago

Thanks! I will give it a try later (hopefully next week).

nao-pon commented 6 months ago

@xen2 I have a question. Only WTY2001 gets uidi from 0xFE, what is the reason? Is the reason for not using f"{uid}-{eojgc}-{eojcc}-{instance}" a change in the instance number of the device managed by this device?

https://github.com/scottyphillips/echonetlite_homeassistant/blob/0ac92848df1bdd6c238cd23187ca3cc6edbe8874/custom_components/echonetlite/config_flow.py#L189-L205

rubyroobs commented 5 months ago

@xen2 I have a question. Only WTY2001 gets uidi from 0xFE, what is the reason? Is the reason for not using f"{uid}-{eojgc}-{eojcc}-{instance}" a change in the instance number of the device managed by this device?

https://github.com/scottyphillips/echonetlite_homeassistant/blob/0ac92848df1bdd6c238cd23187ca3cc6edbe8874/custom_components/echonetlite/config_flow.py#L189-L205

Hi @xen2 - I wanted to bump @nao-pon's question here. I use the same lights and this change required me to reconfigure all my lights after updating to use this to get the uidi. It is neat that the name comes directly from what you set in the Link Plus app, but is there a particular reason the uidi itself needs to be different? It would be nice to confirm so there can be consistent behavior over updates :D

xen2 commented 5 months ago

Sorry for late answer and that it broke your config.

Let me explain why I did this. So far, most device had only 1 instance, and using device uid was enough. But in this case, each switch is not a device but an instance in a device, and this list can be changed in the Panasonic app: if you add a new switch, or simply reorder some switches.

For example, let's say you start with:

Kitchen
Living 1
Living 2

Then you add a new switch (or swap one that was further in the list) in the Panasonic app and let's say it becomes:

Kitchen
Dining Table
Living 1
Living 2

Since there is no way to track those changes (we only have an index), we end up with this in HA:

Kitchen (good, still at index 0)
Living 1 (which actually change dining table, since it now is at index 1)
Living 2 (which actually change living 1, since it now is at index 2)
New light (which will change Living 2) as you can see, it's very confusing.

So my idea was simply to use a proper unique HW ID for each switch and store it in the combined device+instance UID. That way, even if the switch order is swapped/changed, everything is properly tracked.

All of this was not necessary so far since each echonet device usually only had a single instance (UID of device is enough to track changes) and instance were only used for very static stuff (i.e. 2 solar panel instance in a solar panel device). But for the Panasonic switches, it's a single device and the switch list should be considered dynamic (just like a device list). I think it's important to consider the case where one is removed or added, and track it like we usually do for devices themselves.

PS: happy to hear I am not the only one using the panasonic switches. If you are interested, I also worked on a replacement system for their (horrible) app to create and edit groups and scene. It even allows stuff not possible originally (i.e. add or ignore specific lights in a scene even if not part of the group -- this avoid having to create a single group for each different light combination needed).

Here's a screenshot of scene editing: group list (with create button):

Forgive the ugly UI, I mostly focused on the functional part. (I just wished Panasonic scenes/groups could be edited directly on the WTY2001 LAN device, but it has to go through their cloud stuff and then it sync back almost instantly on the WTY2001 device)

I have also created a HA plugin to list groups & scenes with their names (using API exposed by scene/group editing app), and trigger scene in HA using the echonet lite plugin (triggering scene is 100% LAN, it doesn't rely on internet) Here are the scenes grouped in a HA "service" (contain all the scenes applying to a given group of lights)

rubyroobs commented 5 months ago

Sorry for late answer and that it broke your config.

@xen2 no problem at all! Thank you for the very detailed explanation and contributing the change in the first place. I agree this makes sense to switch over to the hardware based IDs for the switches in the way you configured. @nao-pon if you're ok with this too, can we change the edge branch back to using this?

Separately, thanks for sharing your work and sorry to hear you have had the experience of using their switch app also 🤣 For me, as soon as I got them added (which was enough of a pain by itself with their convoluted "工事メニュー" and holding the button every 5 minutes etc) I just controlled everything in HA and made groups in there. I'd honestly be interested in outright replacing the WTY2001 with something running ESPHome or similar but AFAIK there is no work done to reverse engineer their protocol (and bluetooth is not ideal for latency). It's a shame that Lutron etc do not sell in Japan - I had to run so many neutral wires installing my Link Plus switches anyway I would have rather just done that for all of them and used something better...

nao-pon commented 5 months ago

@xen2 , @rubyroobs, I got it. In Panasonic's Link Plus system, instance numbers are sorted in alphabetical order. (Unfortunately) Well, let's adopt your idea! Thanks! 👍

nao-pon commented 3 months ago

This has been fixed in version 3.8.0 and will be closed. Thanks! 👍

scottyphillips / echonetlite_homeassistant

Electric board channels: multiple values in an array #117