Is there a way to get window/door sensor readings?

r0ckarong commented 8 months ago

I've successfully added the exporter to my Synology NAS and now am monitoring the state of my heating and PV system. Thanks for this awesome tool!

One thing I'm missing at the moment and can't figure out how to do is to get the states of my window sensors that govern the heating in various rooms. The devices are listed in the prometheus data but there doesn't seem to be data on their state other than "present".

fritz_ha_device_present{ain="11934 0323032", device_id="UNKNOWN", device_name="Fenster Toilette", friendly_name="Fritz!Box", instance="fritz-exporter:9787", job="fritz-exporter", manufacturer="HAN-FUN", serial="989BCB5F8FE4"} | 2
fritz_ha_device_present{ain="11934 0323032-1", device_id="UNKNOWN", device_name="Fenster Toilette", friendly_name="Fritz!Box", instance="fritz-exporter:9787", job="fritz-exporter", manufacturer="HAN-FUN", serial="989BCB5F8FE4"} | 2
fritz_ha_device_present{ain="11934 0324621", device_id="UNKNOWN", device_name="Fenster Badezimmer", friendly_name="Fritz!Box", instance="fritz-exporter:9787", job="fritz-exporter", manufacturer="HAN-FUN", serial="989BCB5F8FE4"} | 2
fritz_ha_device_present{ain="11934 0324621-1", device_id="UNKNOWN", device_name="Fenster Badezimmer", friendly_name="Fritz!Box", instance="fritz-exporter:9787", job="fritz-exporter", manufacturer="HAN-FUN", serial="989BCB5F8FE4"} | 2
fritz_ha_device_present{ain="11934 0335913", device_id="UNKNOWN", device_name="Terrassentür Küche", friendly_name="Fritz!Box", instance="fritz-exporter:9787", job="fritz-exporter", manufacturer="HAN-FUN", serial="989BCB5F8FE4"} | 2
fritz_ha_device_present{ain="11934 0335913-1", device_id="UNKNOWN", device_name="Terrassentür Küche", friendly_name="Fritz!Box", instance="fritz-exporter:9787", job="fritz-exporter", manufacturer="HAN-FUN", serial="989BCB5F8FE4"}

I would love to get the state "Open / Closed" readings from these sensors to overlay on my Grafana temperature panels. Can you give me an idea what I need to look out for in the API specs then I could try and write some code to do this.

r0ckarong commented 8 months ago

I've dug around in the code and the AVM docs for a while now and I can't find corresponding data for the sensor states. I tried the DECT and Homeauto services but to no avail.

pdreker commented 8 months ago

As discussed in the issue of the original feature request #224 Fritz!OS does not seem to expose some information through the TR-064 API. The windows sensors are simply not provided by AVM (according to the documentation and my quick tests). Battery state of battery powered devices is also not reported, which to me seems like the most inconvenient omission.

I'm sorry, but the device simply does not provide the metrics via the API this exporter uses :( . In #224 there is a mention of another exporter which uses the http API, which provides a different set of data points. Maybe a combination of these two exporters can capture everything.

I will have to make a note of this in the documentation.

pdreker commented 8 months ago

I have made the docs a little more explicit that window sensors and battery state are missing due to an omission from the API.

I'll close this issue, as currently there is nothing I can do about this other than waiting for AVM to add these metrics or actually implementing HTTP access, which is an undocumented API and quite a lot of work.

pdreker commented 8 months ago

closed

r0ckarong commented 8 months ago

which is an undocumented API and quite a lot of work.

I've written to AVM support and they helpfully pointed me to this doc which was published only recently (18th September 23):

https://avm.de/fileadmin/user_upload/Global/Service/Schnittstellen/AHA-HTTP-Interface.pdf

It contains the full HTTP API spec (for home automation) and after some testing I can indeed get all the missing stuff from there by using the fritzconnection library that you're already employing.

I'd be happy to collaborate on implementing that stuff into the exporter if you're up for it.

r0ckarong commented 8 months ago

Initial findings:

Sensors can be accessed through AHA-HTTP Interface

HAN-FUN Sensors hide behind the AIN of their device and the "-1" duplicate (that's where the state data is) For the telekom window sensors there are two states (open and closed)

You can access this data with the getdeviceinfos action and the AIN of the device.

The important bits here is the <alert> tag. This contains the state and the timestamp (since epoch) of the last state change.

The states are:

0 - Closed 1 - Open

.Sensor state "open"

...
<alert><state>1</state><lastalertchgtimestamp>1699253655</lastalertchgtimestamp></alert>
...

{'content': '<device identifier="11934 0323032" id="408" functionbitmask="1" '
            'fwversion="31.35" manufacturer="0x0feb" '
            'productname="HAN-FUN"><present>1</present><txbusy>0</txbusy><name>Fenster '
            'Toilette</name></device>\n',
 'content-type': 'text/xml',
 'encoding': 'utf-8'}

{'content': '<device identifier="11934 0323032-1" id="2001" '
            'functionbitmask="8208" fwversion="0.0" manufacturer="0x0feb" '
            'productname="HAN-FUN"><present>1</present><txbusy>0</txbusy><name>Fenster '
            'Toilette</name><etsiunitinfo><etsideviceid>408</etsideviceid><unittype>514</unittype><interfaces>256</interfaces></etsiunitinfo><alert><state>1</state><lastalertchgtimestamp>1699253655</lastalertchgtimestamp></alert></device>\n',
 'content-type': 'text/xml',
 'encoding': 'utf-8'}

.Sensor state "closed"

...
<alert><state>0</state><lastalertchgtimestamp>1699259643</lastalertchgtimestamp></alert>
...

{'content': '<device identifier="11934 0351393" id="411" functionbitmask="1" '
            'fwversion="31.35" manufacturer="0x0feb" '
            'productname="HAN-FUN"><present>1</present><txbusy>0</txbusy><name>Tür '
            'Kinderzimmer </name></device>\n',
 'content-type': 'text/xml',
 'encoding': 'utf-8'}

{'content': '<device identifier="11934 0351393-1" id="2003" '
            'functionbitmask="8208" fwversion="0.0" manufacturer="0x0feb" '
            'productname="HAN-FUN"><present>1</present><txbusy>0</txbusy><name>Tür '
            'Kinderzimmer '
            '</name><etsiunitinfo><etsideviceid>411</etsideviceid><unittype>514</unittype><interfaces>256</interfaces></etsiunitinfo><alert><state>0</state><lastalertchgtimestamp>1699259643</lastalertchgtimestamp></alert></device>\n',
 'content-type': 'text/xml',
 'encoding': 'utf-8'}

Thermostats

The Thermostats (at least Fritz ones) can also be accessed via their AIN via AHA-HTTP. This exposes all kinds of data including battery.

tsoll, tist, komfort are the temperature values that are set with the programmed schedule. NOTE: They are for some reason multiplied by 2 over what is set in the UI (probably to avoid having floats and the UI only allows half degree steps).

Weirdly the <temperature> (measured temperature) is multiplied by 10.

lock and devicelock the operation locks. Need to determine further which is which (one is button lock and the other is overall operations lock).

windowopenactive shows if the window detection is triggered (probably also by associated sensor, need to test)

Basically all the info from the FritzUI is exposed here.

.Fritz Dect 301 device (formatted)

{'content': '<device identifier="09995 0390185" id="16" functionbitmask="320" '
            ' fwversion="05.08" manufacturer="AVM" productname="FRITZ!DECT '
            '301">
  <present>1</present>
  <txbusy>0</txbusy>
  <name>Wohnzimmer '
    'Süd</name>
  <battery>80</battery>
  <batterylow>0</batterylow>
  <temperature>
    <celsius>195</celsius>
    <offset>0</offset>
  </temperature>
  <hkr>
    <tist>39</tist>
    <tsoll>30</tsoll>
    <absenk>30</absenk>
    <komfort>42</komfort>
    <lock>0</lock>
    <devicelock>1</devicelock>
    <errorcode>0</errorcode>
    <windowopenactiv>0</windowopenactiv>
    <windowopenactiveendtime>0</windowopenactiveendtime>
    <boostactive>0</boostactive>
    <boostactiveendtime>0</boostactiveendtime>
    <batterylow>0</batterylow>
    <battery>80</battery>
    <nextchange>
      <endperiod>1699290000</endperiod>
      <tchange>42</tchange>
    </nextchange>
    <summeractive>0</summeractive>
    <holidayactive>0</holidayactive>
    <adaptiveHeatingActive>1</adaptiveHeatingActive>
    <adaptiveHeatingRunning>0</adaptiveHeatingRunning>
  </hkr>
</device>\n',
'content-type': 'text/xml',
'encoding': 'utf-8'}

TODO

. Implement device discovery from HTTP API . Find list of sensors . Write function to generate metrics from the sensor data . Write logic that treats this data like all the other data . Modify the access helper to work in HTTP mode with parameters and allows looking at singular devices

Nice to have: . Decipher how the energy data stats for the DECT 200/210 are used + .getbasicdevicestats for a DECT 210

{'content': '<devicestats><temperature><stats count="96" grid="900" '
            'datatime="1699286551">110,110,110,115,115,115,115,115,115,120,120,125,130,130,125,125,125,130,125,120,120,120,115,115,115,110,110,110,115,110,110,110,110,110,105,105,105,105,105,105,105,105,105,105,105,105,105,105,105,110,105,105,105,105,110,110,105,105,105,105,105,105,105,105,105,105,105,105,105,105,105,105,105,105,105,105,105,105,110,100,105,100,105,105,105,105,105,105,105,105,105,110,110,110,110,115</stats></temperature><voltage><stats '
            'count="360" grid="10" '
            'datatime="1699286551">235333,235333,235333,235333,235333,235333,235333,235333,235333,235333,235333,235333,234841,234841,234841,234841,234841,234841,234841,234841,234841,234841,234841,234841,235530,235530,235530,235530,235530,235530,235530,235530,235530,235530,235530,235530,235606,235606,235606,235606,235606,235606,235606,235606,235606,235606,235606,235606,235606,235754,235754,235754,235754,235754,235754,235754,235754,235754,235754,235754,235754,235645,235645,235645,235645,235645,235645,235645,235645,235645,235645,235645,235645,235462,235462,235462,235462,235462,235462,235462,235462,235462,235462,235462,235462,235494,235494,235494,235494,235494,235494,235494,235494,235494,235494,235494,235494,235750,235750,235750,235750,235750,235750,235750,235750,235750,235750,235750,235750,235736,235736,235736,235736,235736,235736,235736,235736,235736,235736,235736,235736,235695,235695,235695,235695,235695,235695,235695,235695,235695,235695,235695,235841,235841,235841,235841,235841,235841,235841,235841,235841,235841,235841,235841,235255,235255,235255,235255,235255,235255,235255,235255,235255,235255,235255,235255,235419,235419,235419,235419,235419,235419,235419,235419,235419,235419,235419,235419,235240,235240,235240,235240,235240,235240,235240,235240,235240,235240,235240,235240,235209,235209,235209,235209,235209,235209,235209,235209,235209,235209,235209,235209,235492,235492,235492,235492,235492,235492,235492,235492,235492,235492,235492,235492,235646,235646,235646,235646,235646,235646,235646,235646,235646,235646,235646,235646,235456,235456,235456,235456,235456,235456,235456,235456,235456,235456,235456,235456,235665,235665,235665,235665,235665,235665,235665,235665,235665,235665,235665,235665,235397,235397,235397,235397,235397,235397,235397,235397,235397,235397,235397,235397,235275,235275,235275,235275,235275,235275,235275,235275,235275,235275,235275,235275,235771,235771,235771,235771,235771,235771,235771,235771,235771,235771,235771,235771,235877,235877,235877,235877,235877,235877,235877,235877,235877,235877,235877,235877,235882,235882,235882,235882,235882,235882,235882,235882,235882,235882,235882,235882,236150,236150,236150,236150,236150,236150,236150,236150,236150,236150,236150,236150,236143,236143,236143,236143,236143,236143,236143,236143,236143,236143,236143,236143,236183,236183,236183,236183,236183,236183,236183,236183,236183,236183,236183,236183,235572,235572,235572,235572,235572,235572,235572,235572,235572,235572,235572,235572,236118,236118,236118,236118,236118,236118,236118,236118,236118,236118,236118,236118</stats></voltage><power><stats '
            'count="360" grid="10" '
            'datatime="1699286551">14,14,14,14,14,14,14,14,14,14,14,14,13,13,13,13,13,13,13,13,13,13,13,13,14,14,14,14,14,14,14,14,14,14,14,14,13,13,13,13,13,13,13,13,13,13,13,13,13,13,13,13,13,13,13,13,13,13,13,13,13,14,14,14,14,14,14,14,14,14,14,14,14,15,15,15,15,15,15,15,15,15,15,15,15,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,181,181,181,181,181,181,181,181,181,181,181,181,345,345,345,345,345,345,345,345,345,345,345,345,578,578,578,578,578,578,578,578,578,578,578,578,746,746,746,746,746,746,746,746,746,746,746,746,529,529,529,529,529,529,529,529,529,529,529,529,523,523,523,523,523,523,523,523,523,523,523,523,476,476,476,476,476,476,476,476,476,476,476,476,444,444,444,444,444,444,444,444,444,444,444,444,384,384,384,384,384,384,384,384,384,384,384,384,439,439,439,439,439,439,439,439,439,439,439,439,472,472,472,472,472,472,472,472,472,472,472,472,488,488,488,488,488,488,488,488,488,488,488,488,537,537,537,537,537,537,537,537,537,537,537,537,577,577,577,577,577,577,577,577,577,577,577,577,597,597,597,597,597,597,597,597,597,597,597,597,693,693,693,693,693,693,693,693,693,693,693,693,621,621,621,621,621,621,621,621,621,621,621,621,624,624,624,624,624,624,624,624,624,624,624,624,625,625,625,625,625,625,625,625,625,625,625,625</stats></power><energy><stats '
            'count="12" grid="2678400" '
            'datatime="1699232408">3119,9126,0,0,0,0,0,0,0,0,0,0</stats><stats '
            'count="31" grid="86400" '
            'datatime="1699232407">838,267,438,609,189,778,533,419,136,186,680,435,137,132,1022,505,1536,309,264,1712,1120,0,0,0,0,0,0,0,0,0,0</stats></energy></devicestats>\n',
 'content-type': 'text/xml',
 'encoding': 'utf-8'}

r0ckarong commented 8 months ago

I did all the testing by doing some very basic modifications to the fritz_export_helper.py. All it needs is using call_http instead of call_action to the authenticated connection and then providing the correct action (and optional AIN) from the API doc. The sid is handled automatically by fritzconnection.

pdreker commented 8 months ago

Holy cow, Bat-Man!

Hadn't checked for quite some time, but this is some really nice information, especially if this is usable through fritzconnection.

I'll have a look towards the weekend, but obviously this will be re-opened :-D

Additionally this might solve the "No Cable Data" issues.

r0ckarong commented 8 months ago

Additionally this might solve the "No Cable Data" issues.

From a first glance the API does not provide anything related to the network or internet connections. AHA stands for "AVM Home Automation" and the data is entirely related to smart home devices and their control (which is awesome anyway).

r0ckarong commented 8 months ago

https://fritzconnection.readthedocs.io/en/latest/sources/getting_started.html#combining-the-apis

pdreker commented 6 months ago

It wasn't the next weekend, but here we are, with (maybe) a little present for the holidays :-)

I have just pushed some changes to the develop version, which read information from devices via the HTTP AHA API. Currently it only checks every device reported by the box for "battery" and "battery_low" values and converts them into metrics if found.

As I myself do not have any of the HA devices I cannot test those changes. Could you have a got with the current "develop" image and check, if you can see fritz_ha_battery_level_percentand fritz_ha_battery_low metrics?

If this works I can extend the metrics reported relatively easily.

r0ckarong commented 6 months ago

Oh look, open source Santa dancing on my roof. 😄

I'll try have a look at this in the coming days.

r0ckarong commented 6 months ago

So I finally managed to make some time to test and the develop container crashes immediately after start for me:

2024-01-03 20:11:44,508     INFO fritzexporter.fritzdevice | Connection to 192.168.178.1 successful, reading capabilities
2024-01-03 20:11:47,111     INFO fritzexporter.fritzdevice | Reading capabilities for 192.168.178.1, got serial <REDACTED> , model name FRITZ!Box 7530 completed
2024-01-03 20:11:47,111     INFO fritzexporter | registering 192.168.178.1 to collector
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/xml/etree/ElementTree.py", line 1701, in close
    self.parser.Parse(b"", True) # end of data
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
xml.parsers.expat.ExpatError: no element found: line 1, column 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/local/lib/python3.12/site-packages/fritzexporter/__main__.py", line 126, in <module>
    main()
  File "/usr/local/lib/python3.12/site-packages/fritzexporter/__main__.py", line 109, in main
    REGISTRY.register(fritzcollector)
  File "/usr/local/lib/python3.12/site-packages/prometheus_client/registry.py", line 40, in register
    names = self._get_names(collector)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/prometheus_client/registry.py", line 80, in _get_names
    for metric in desc_func():
  File "/usr/local/lib/python3.12/site-packages/fritzexporter/fritzdevice.py", line 105, in collect
    yield from capa.get_metrics(self.devices, name)
  File "/usr/local/lib/python3.12/site-packages/fritzexporter/fritzcapabilities.py", line 87, in get_metrics
    self._generate_metric_values(device)
  File "/usr/local/lib/python3.12/site-packages/fritzexporter/fritzcapabilities.py", line 1413, in _generate_metric_values
    http_data = parse_aha_device_xml(http_result["content"])
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/fritzexporter/fritz_aha.py", line 5, in parse_aha_device_xml
    device: ElementTree = ElementTree.fromstring(deviceinfo)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/defusedxml/common.py", line 127, in fromstring
    return parser.close()
           ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/xml/etree/ElementTree.py", line 1703, in close
    self._raiseerror(v)
  File "/usr/local/lib/python3.12/xml/etree/ElementTree.py", line 1603, in _raiseerror
    raise err
xml.etree.ElementTree.ParseError: no element found: line 1, column 0

r0ckarong commented 6 months ago

Btw. there is super useful but undocumented overview for all the DECT Smart Home devices: http://fritz.box/#uleoverview

r0ckarong commented 6 months ago

I did some more testing and I think the problem is that I have non AVM devices in my setup that do not report battery stats and therefore there is no "data" in the result for this check on such a device.

HAN-FUN device

{'content': '<device identifier="11934 0324621-1" id="2000" '
            'functionbitmask="8208" fwversion="0.0" manufacturer="0x0feb" '
            'productname="HAN-FUN"><present>1</present><txbusy>0</txbusy><name>Fenster '
            'Badezimmer</name><etsiunitinfo><etsideviceid>407</etsideviceid><unittype>514</unittype><interfaces>256</interfaces></etsiunitinfo><alert><state>0</state><lastalertchgtimestamp>1704277584</lastalertchgtimestamp></alert></device>\n',

AVM device

{'content': '<device identifier="09995 0387424" id="20" functionbitmask="320" '
            'fwversion="05.08" manufacturer="AVM" productname="FRITZ!DECT '
            '301"><present>1</present><txbusy>0</txbusy><name>Schlafzimmer '
            </name><battery>70</battery><batterylow>0</batterylow><temperature><celsius>180</celsius><offset>0</offset></temperature><hkr><tist>36</tist><tsoll>38</tsoll><absenk>32</absenk><komfort>38</komfort><lock>0</lock><devicelock>0</devicelock><errorcode>0</errorcode><windowopenactiv>0</windowopenactiv><windowopenactiveendtime>0</windowopenactiveendtime><boostactive>0</boostactive><boostactiveendtime>0</boostactiveendtime><batterylow>0</batterylow><battery>70</battery><nextchange><endperiod>1704320100</endperiod><tchange>32</tchange></nextchange><summeractive>0</summeractive><holidayactive>0</holidayactive><adaptiveHeatingActive>1</adaptiveHeatingActive><adaptiveHeatingRunning>0</adaptiveHeatingRunning></hkr></device>\n',

I guess the easiest fix is to, for now, limit the check to manufacturer="AVM" for the battery.

There probably are plenty of devices out there that report interesting metrics. My HAN-FUN sensors for example give me their state and when they last triggered. That could be interesting to visualize with the temperature changes etc.

I am still thinking about how to make this easier to integrate than you having to hardcode any metrics and filter for all types of device manufacturers. My first idea was to somehow make the metrics and sources configurable outside the code but this takes some more thinking.

My suggestion would be to stick to the "known" AVM devices and their reported stuff and integrate those metrics.

pdreker commented 5 months ago

In the past I quite literally stuck to the Python motto of "It is better to ask foregiveness than permission." In this context this means that I would go for a variant, which just tries to "go for it" and if for example the XML Parsing ocde throws an exception like above, I just catch that exception, log a (DEBUG level?) message and just ignore the problem and not add a metric there.

This avoids the whole hardcoding IDs problem and still works whenever a device returns a battery state. There is an example in the capabilities check, where it seems that some device just report endpoint they are actually not supporting. I just catch the FritzServiceError (and similar) and remove the metric. In this case it would be more prudent to do something like this when actually fetching the metrics from the device, instead when chacking capabilities.

pdreker / fritz_exporter