search

emsesp / EMS-ESP32

ESP32 firmware to read and control EMS and Heatronic compatible equipment such as boilers, thermostats, solar modules, and heat pumps
https://emsesp.github.io/docs
GNU Lesser General Public License v3.0
631 stars 107 forks source link

KM200 gateway vs. ems-esp: Would it make sense to harmonize data structure - or integrate http interface to km200? #30

Closed tp1de closed 3 years ago

tp1de commented 3 years ago

I am using the rest-api interface of the km200 internet gateway since 2 1/2 years to integrate the heating system into my home automation system (ioBroker). The interface is not documented and a bit complex since it is encrypted (rinjdael-128 ecb) and every value has to be polled seperate. In my case with KB192i boiler, rc310 thermostat and mm100 mixer and 2 heating circuits, I will get approx 170 datapoints back. The polling of the km200 is very slow and and lot of http get-request have timeouts. So I bought the ems-esp to get better realtime data for hydraulic optimization of my system. This is working excellent.

Once using both system it is easy to recognize that the data-structure and the data-names are quite different. So it is not trivial to find 1:1 the respective values. The km200 structure is not per device but more per function: grafik Would it make sense midterm to harmonize data-structure and content?

There are some adition datapoints in the ems-esp not known by km200 (yet) -> WW circulation, Mixer valve status etc Since there are automatic firmware updates every 3-4 months the km200 datatpoints are slightly changing over time..

A very interesting content is within recordings. There are time recordings on energy-consumption (heating and ww) and temperatures available. (per day, per month, per year). These values are quite accurate - I implemented a correction factor towards the gas-meter. My multiplier is 1.046 and over the year my difference is less then 10kWh.

I believe that these values might be of interest for everybody who has a boiler which provides this data. A very easy way could be to implement http-request handling and decryption on the esp software. @MichaelDvP you might be the expert for this. Or we need to search for the respective telegrams ... please tell me where I can support.

For refererence I will attach my km200 data structure: km200.0.json.zip

I would be very interested to get your feedback.

Best regards Thomas

proddy commented 3 years ago

I prefer to keep EMS-ESP to just decoding of the EMS bus messages, and not using HTTP to query the devices. It's adds a lot of complexity to the software and quite frankly the ESP8266 isn't powerful enough to handle this, and there are better suited solutions to this (like a Python script in HA as we do with the Easy thermostats).

However if the KM200 is capturing this somehow then I'm hoping it's also exposing this data via EMS telegrams. What we should try is brute-force read request on the whole telegram range (0->0xFFFF) and see what comes back?

I'm looking at re-writing the Web server for the next generation ESP32 release which could potentially make HTTP/S calls easier, but that's something for the future.

tp1de commented 3 years ago

@proddy > I prefer to keep EMS-ESP to just decoding of the EMS bus messages, and not using HTTP to query the devices.

understood. What about harmonizing data-structure and names of datapoints? (midterm) By the way is there any documentation available which ems-esp datapoints are writeable?

glitter-ball commented 3 years ago

I’d also wondered about a ‘port probe’ telegram read. Doing [almost] random reads was how we found the damped outside temperature in the CW400 / Sense II which wasn’t published.

Could do with the raw telegrams then parsed by the common number formats into decimal values for humans to guess what they might be.

I guess a bulk read isn’t going to break anything so not much to lose (other than time...)

proddy commented 3 years ago

understood. What about harmonizing data-structure and names of datapoints? (midterm) By the way is there any documentation available which ems-esp datapoints are writeable?

@tp1de the datapoints are just the names of the devices (solar, thermostat, boiler) but they do differ depending on the context. For example with MQTT they are prefixed to keep the payload size small (boiler_ww, boiler_data etc). And in the HTTP API it's like http://ems-esp/api?device=boiler&cmd=info. Write commands are done via commands, and again that differs per device. You can see this all in the documentation (https://emsesp.github.io/docs/#/API).

It's just kinda evolved this way, based on limitations of ESP8266 microcontroller (TCP buffer size, speed, memory...). I'm open to fresh new ideas and moved exclusively to the ESP32 for further expansion.

So are you suggesting for example normalize into heatingSoure, heatingCircuits as the gateway does? If there would be a mapping on how this would look like for each EMS-ESP device, not forgetting the mixers and solar modules I'd be happy to consider a move.

MichaelDvP commented 3 years ago

I've looked into the structure of the ioBroker-adapter, reading the KM200 here. The logic how parameters sorted to heatSources, heatSources/hs1, system/heatSources/hs1 etc. are not clear to me, mixers and solar is missing there.

The recordings needs a lot of memory, i suppose them as function of the bosch-cloud and not integrated.

For the parameter names i think we wait for emsesp/EMS-ESP32#22, then it should be easy to add and select a compatible nameset.

tp1de commented 3 years ago

@MichaelDvP

The recordings needs a lot of memory, i suppose them as function of the bosch-cloud and not integrated.

The data poll is done by local http request to km200. I think that the recordings are calculated by the mc110 master controller.

I 've looked into the structure of the ioBroker-adapter, reading the KM200 here. The logic how parameters sorted to heatSources, heatSources/hs1, system/heatSources/hs1 etc. are not clear to me, mixers and solar is missing there.

The rest api has no possibility to return a complete datastructure. So every datapoint requires a seperate http get request. To find all available points and encounter the datastructure the km200 adapter starts at destinctive entry points. Thea are preset by experience since documentation is available from Bosch.

On http get ..../heatSources the encrypted datastream returns decrypted an array of datapoints ... in my case 29. The array contains an id and and uri-adress of the next level. Making a http request there km200 either returns an object - e.g.:

object id: "/heatSources/actualSupplyTemperature" type: "floatValue", writeable: 0,recordable: 0,value: 50.7,unitOfMeasure: "C"

or another array for next level.

The main entry points are:

The heatingCircuits datapoints contain switchPrograms, Summer/Winter mode, temperature levels etc. There is no direct indication if a mm100 mixer module is attached to the hc. The heating control parameters are not readable by the interface. A floorheating with mixer can only be identified by lower temp set-points....

Every 2-4 month a new firmware for the km200 is updated by Bosch and datapoints slightly change. In Sep 20 the return temp disappeared and end of December it came back again. The circulation pump for ww is not integrated yet but promised that the respective functionality will be added.

The iobroker km200 adapter reads the defined main entry points and then reads the whole structure down. This is done by quite a lot of http request until the whole structure is known. As any other adapter in iobroker the adapter then creates the datapoints in the iobroker data-repository. When values are changed in iobroker, then per http post the values are updated in km200.

I basically use node-red to read and update. Both ways are possible.

MichaelDvP commented 3 years ago

The data poll is done by local http request to km200. I think that the recordings are calculated by the mc110 master controller.

Can you log the telegrams between KM200 and MC110 while requesting the recodings from KM200. Then we should see the telegram structure and can add these telegrams to ems-esp.