Rework parser: Use function IDs instead of device IDs to determine functionality

Summary

I've reworked the XML parser to determine device functionality based on function IDs. This should be much more robust than using device IDs, and newer devices should be supported automatically.

This PR is still a work in progress. Feedback is welcome, and I'd appreciate your help in adding support for more hardware (see "Next steps").

Related: #39

Changes

Get full XML representation: send_rpc_iq('RemoteInterface.getAll', 'de', 4, 0, 0) only returns a partial XML representation. I guess only the parts relevant for the SysAP control UI are returned. By using send_rpc_iq('RemoteInterface.getAll', 'de', 2, 0, 0) we get a complete XML that includes much more information, e.g. all channels for binary sensors with their initial state. My guess is that this information is loaded by the SysAP admin UI, although it also works with a restricted user.
Rework parser: Instead of checking device IDs, we now iterate over all channels of all devices, ignore those that are not assigned to a room (i.e. without functionality) and derive their function from the function ID (e.g. light actuator, blind actuator, sensor)
Group entities in the devices panel in Home Assistant: The devices now show up in HA, with some additional information:
Hue and other external actuators are now ignored, since they can be directly integrated into Home Assistant
Fixed get-master-message.py to work with newer Python version as well as async XMPP code. The command now also saves a pretty-printed XML file that should be better readable.

Working

I've tested the code with the following devices in different configurations, and everything works as expected:

1000: Sensor 1x
1002: Sensor 2x
1004: Room temperature controller
100C: Sensor / switch actuator 1x
1010: Sensor / switch actuator 2x
1013: Sensor / blind actuator
B001: Blind actuator 4x REG
B004: Heating actuator 12x REG (controlled through thermostats)
B008: Sensor / switch actuator 8x REG

Also other devices that provide roughly the same functionality should work (e.g. Heating actuator 8x)

Not yet working

Due to lack of hardware, the following devices are not yet supported by this code:

Binary sensors (2x or 4x, e.g. 6241/2.0U) (or maybe they'll work out of the box? I can't tell.)
Movement detector
Weather station
Lock (via Panel)

Next steps

Add support for more hardware (see above). If you own such a device, it would be nice if you could send me the mastermessage.xml that can be generated by running python get-master-message.py. I can probably derive correct function IDs from it and add them to the code.
Determine correct input/output datapoints either from their nameId, or from their pairingId, as suggested in #39

I'm looking into movement detectors right now, and the more I read my mastermessage.xml, the more I think we definitely have to take pairingIds of input and output datapoints into account.

Take for example this getall.xml. If you search for 900A, you will find a movement detector device. The first channel (functionId 11) is the actual movement detector, and according to the current code, the correct output datapoint would be odp0000. So the device will write 1 to odp0000 if movement is detected.

Now on the other hand, take a look at this snippet of my mastermessage.xml This is the first channel of an 8-gang sensor/actuator unit. You can attach different kinds of sensors to the inputs, for example a regular switch as a binary input, or even a movement detector. But, if you configure it with a movement detector, the actual output datapoint will be odp000D, named Presence. So if we want this component to detect this unit correctly, we will have to correlate a datapoint's pairingId with the intended function.

I'll investigate further.

Do you have access to the developer portal of free@home? Documentation might help you in analysis. Basically channels etc are documented. You can register fast and easy.

Also great that Hue will be ignored. Since it indeed works in HA directly but with the Hue emulator in Home Assistant you can also expose to Free@Home. Previously this did not work but the component got improved. This means that it makes sense to actually put home assistent in the center of the universe and then smart bulbs start performing like a charm...

The FAH api also has the ability to emulate functionality hardware to the sysap, this might also be something nice to implement later on. Can also be found in the documentation.

Thumbs up! If this is somewhat stable in changes I can test it, but really depending on Home Assistant already so if functionality is quite stable during the refactoring might be nice :).

Do you have access to the developer portal of free@home? Documentation might help you in analysis. Basically channels etc are documented. You can register fast and easy.

I have seen this, it is actually open to anyone without registration here. Good resource, although the mapping between function IDs, pairing IDs and output datapoints still seems to be undocumented. I guess this will have to be worked out by analyzing config files.

Also great that Hue will be ignored. Since it indeed works in HA directly but with the Hue emulator in Home Assistant you can also expose to Free@Home. Previously this did not work but the component got improved. This means that it makes sense to actually put home assistent in the center of the universe and then smart bulbs start performing like a charm...

Yes, guess who actually did the improvement to emulated_hue :wave: :wink: I also have this kind of two-way integration, really useful. Although controlling dimmers defined in HA through free@home in-wall switches is laggy. I guess this is due to the Hue implementation on the SysAP.

Thumbs up! If this is somewhat stable in changes I can test it, but really depending on Home Assistant already so if functionality is quite stable during the refactoring might be nice :).

I can relate, I don't actually run this on my main instance yet. That's why I setup a second HA instance on my desktop PC (Docker), so that I can develop while my house continues to work. If you could provide your configuration (either through the get-master-message.py from this branch, or from a .pro backup file), I could look through it and maybe add support for more devices. If you want to send them by email you may find my address in one of the commits.

auto backup 20201012 1853-202010121853.pro.zip see attached...

Thumbs up for all the efforts. I have a lack of time on getting familiar with Python on this one but can provide every way of help here to improve with validating, debugging, etc.

[offtopic] One of the biggest smiles on my face was when I attached the hue emulator with deconz in between and started threading groups as single lights within FAH. All lights on at once instead of one by one with lag... lovely :). [/offtopic]

And to confirm... Dimming with the switches is indeed laggy. At my side it already improved when I upgraded from previous version of the SysAP hardware to the latest so that (for me) confirms your thoughts about the implementation at FAH side.

The binary sensor support should now be complete. I tested it with my in-wall switches as well as with a binary sensor 2-gang unit (6241/2.0 U) in all configurations, and I can report that the following configurations work as expected:

Switch sensor
Dimming sensor
Force on/off sensor
Cover force position sensor
Switch heating/cooling
Wind alarm
Frost alarm
Rain alarm
Window contact

The following ones do not work as intended, at least with my binary sensor 2-gang unit:

Cover sensor: Reports as "off" when moving up, or "on" when moving down. This persists after the cover has stopped
Scene sensor: This does not report any datapoint to HA, so there's nothing we can react to
Movement detector: As outlined in #37, movement detectors only report as "on" once and do not change back to "off" after movement has stopped. This is a limitation in free@home, and there's nothing we can do about it :-( This may not be a problem for dedicated movement detectors, because they seem to emit another datapoint called "presence"
Staircase sensor aka timed on/off: Same as movement detector, only reports once as "on"

The last things on my list are the weather station and the lock. Due to lack of hardware I'll have to add a good guess for pairing IDs and hope someone with this hardware can test it.

I think I'm mostly done here. I've added support for weather stations and the lock, although I could not test them. Please test this branch with your devices and report anything that isn't working. Otherwise, feel free to merge this.

Running this branch now and can confirm that it works perfectly nice! Great step forward. Is there anyone with data for the lock and weather station to improve?

Hi, I tried to use this branch but I found an error. I want to say I'm not using the Busch-Jaeger sysap but I have the ABB one, so something may be different. My sysap is on version 2.5.5.

The error I'm getting is this one:

2020-10-27 23:34:02 ERROR (MainThread) [homeassistant.config_entries] Error setting up entry 192.168.1.10 for freeathome
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 231, in async_setup
    result = await component.async_setup_entry(hass, self)  # type: ignore
  File "/config/custom_components/freeathome/__init__.py", line 75, in async_setup_entry
    await sysap.find_devices()
  File "/config/custom_components/freeathome/fah/pfreeathome.py", line 774, in find_devices
    await self.xmpp.find_devices(self._use_room_names)
  File "/config/custom_components/freeathome/fah/pfreeathome.py", line 683, in find_devices
    datapoints = get_datapoints_by_pairing_ids(channel, pairing_ids)
  File "/config/custom_components/freeathome/fah/pfreeathome.py", line 156, in get_datapoints_by_pairing_ids
    datapoints[pairing_id] = get_datapoint_by_pairing_id(xmlnode, type, pairing_id)
  File "/config/custom_components/freeathome/fah/pfreeathome.py", line 145, in get_datapoint_by_pairing_id
    if int(datapoint.get('pairingId'), 16) == pairing_id:
TypeError: int() can't convert non-string with explicit base

In fact I did some troubleshooting and I verified that datapoint.get('pairingId') returns None and that the datapoint tag doesn't contain any pairingId tag. An example of a thermostat device is this:

<device isTp="true" domainAddress="BAE7" individualAddress="0146" nameId="00EA" profile="0E00" maxAPDULength="39" compilerVersion="00000000" buildNumber="00000000" iconId="FFF6" protocolVersion="0000" minConfigVersion="0000" deviceFlavor="00" interface="TP" functionId="FEFE" shortSerialNumber="MYN" softwareId="0853" softwareVersion="1.83" deviceId="010D" serialNumber="ABB5000XXXXX" commissioningState="ready">
    <channels>
        <channel mask="FFFFFFFF" nameId="0062" i="ch0000" cid="ABB0ABCD">
            <attribute name="displayName">Thermostat</attribute>
            <attribute name="floor">02</attribute>
            <attribute name="functionId">23</attribute>
            <attribute name="offsetX">0.593206592228102</attribute>
            <attribute name="offsetY">0.600826188126858</attribute>
            <attribute name="room">09</attribute>
            <attribute name="selectedIcon">21</attribute>
            <inputs>
                <dataPoint i="idp0005">
                    <value>0</value>
                    <address timeSlot="0" primary="true">a15d</address>
                </dataPoint>
            </inputs>
            <outputs>
                <dataPoint i="odp0000">
                    <value>0</value>
                    <address timeSlot="0" primary="true">5339</address>
                </dataPoint>
                <dataPoint i="odp0002">
                    <value>15</value>
                    <address timeSlot="0" primary="true">6839</address>
                </dataPoint>
                <dataPoint i="odp0003">
                    <value>-6</value>
                    <address timeSlot="0" primary="true">475c</address>
                </dataPoint>
                <dataPoint i="odp0004">
                    <value>1</value>
                    <address timeSlot="0" primary="true">6b3c</address>
                </dataPoint>
                <dataPoint i="odp0005">
                    <value>18.9</value>
                    <address timeSlot="0" primary="true">6981</address>
                </dataPoint>
                <dataPoint i="odp0006">
                    <value>97</value>
                    <address timeSlot="0" primary="true">68b6</address>
                </dataPoint>
                <dataPoint i="odp000B">
                    <value>0</value>
                    <address timeSlot="0" primary="true">265c</address>
                </dataPoint>
                <dataPoint i="odp000C">
                    <value>0</value>
                    <address timeSlot="0" primary="true">692e</address>
                </dataPoint>
            </outputs>
            <parameters>
                <parameter i="pm0000" optional="false">
                    <value>21</value>
                </parameter>
                <parameter i="pm0001" optional="false">
                    <value>3</value>
                </parameter>
                <parameter i="pm0002" optional="false"/>
                <parameter i="pm0003" optional="false"/>
                <parameter i="pm0005" optional="false"/>
            </parameters>
        </channel>
    </channels>
</device>

At this point I tried to figure out how to get pairing ids, and I found that right before the devices tag there is the descriptions tag that seems to associate in some way datapoints to pairing ids. Here is an example for the thermostat:

<descriptions>
    [...]
    <channel channelId="0805" maxScenes="08" cid="ABB0ABCD"> # cid here is the same as the cid in the device before
        <functions>
            <function actuatorMatchCode="00000001" sensorMatchCode="10000000" accessLevel="Enduser" nameId="01BB" isDefault="false" functionId="000A" fixed="false" bestMatch="true"/>
            <function actuatorMatchCode="00000002" sensorMatchCode="20000000" accessLevel="Enduser" nameId="01BA" isDefault="true" functionId="0023" fixed="false" bestMatch="true"/>
        </functions>
        <inputs>
            <dataPoint pairingId="0039" dpt="0901" matchCode="00000003" maxConnections="10" autoConnectHouse="false" autoConnectFloor="false" autoConnectRoom="false" defaultConnection="false" nameId="0115"/>
            <dataPoint pairingId="0042" dpt="0101" matchCode="00000003" maxConnections="10" autoConnectHouse="false" autoConnectFloor="false" autoConnectRoom="false" defaultConnection="false" nameId="0119"/>
            <dataPoint pairingId="003A" dpt="0101" matchCode="00000003" maxConnections="10" autoConnectHouse="false" autoConnectFloor="false" autoConnectRoom="false" defaultConnection="false" nameId="0117"/>
            <dataPoint pairingId="0004" dpt="1201" matchCode="00000003" maxConnections="20" autoConnectHouse="false" autoConnectFloor="false" autoConnectRoom="false" defaultConnection="false" nameId="000E"/>
            <dataPoint pairingId="0140" dpt="0901" matchCode="00000003" pairingOptional="false" maxConnections="01" autoConnectHouse="false" autoConnectFloor="false" autoConnectRoom="false" defaultConnection="false" nameId="0205"/>
            <dataPoint pairingId="0131" dpt="0501" matchCode="30000000" maxConnections="10" autoConnectHouse="false" autoConnectFloor="false" autoConnectRoom="false" defaultConnection="false" nameId="017D"/>
            <dataPoint pairingId="0132" dpt="0501" matchCode="30000000" maxConnections="10" autoConnectHouse="false" autoConnectFloor="false" autoConnectRoom="false" defaultConnection="false" nameId="01C7"/>
            <dataPoint pairingId="014F" dpt="0501" matchCode="00000003" maxConnections="01" autoConnectHouse="false" autoConnectFloor="false" autoConnectRoom="false" defaultConnection="false" nameId="039E"/>
            <dataPoint pairingId="0150" dpt="0501" matchCode="00000003" maxConnections="01" autoConnectHouse="false" autoConnectFloor="false" autoConnectRoom="false" defaultConnection="false" nameId="039F"/>
            <dataPoint pairingId="0035" dpt="0113" matchCode="00000003" maxConnections="10" autoConnectHouse="false" autoConnectFloor="false" autoConnectRoom="false" defaultConnection="false" nameId="012E"/>
            <dataPoint pairingId="0040" dpt="1465" matchCode="00000003" maxConnections="10" autoConnectHouse="false" autoConnectFloor="false" autoConnectRoom="false" defaultConnection="false" nameId="0116"/>
            <dataPoint pairingId="0145" dpt="050A" matchCode="10000000" maxConnections="10" autoConnectHouse="false" autoConnectFloor="false" autoConnectRoom="true" defaultConnection="false" nameId="01F0"/>
            <dataPoint pairingId="0146" dpt="0101" matchCode="10000000" maxConnections="10" autoConnectHouse="false" autoConnectFloor="false" autoConnectRoom="true" defaultConnection="false" nameId="01F1"/>
            <dataPoint pairingId="0149" dpt="050A" matchCode="10000000" maxConnections="10" autoConnectHouse="false" autoConnectFloor="false" autoConnectRoom="true" defaultConnection="false" nameId="02C5"/>
            <dataPoint pairingId="014A" dpt="0101" matchCode="10000000" maxConnections="10" autoConnectHouse="false" autoConnectFloor="false" autoConnectRoom="true" defaultConnection="false" nameId="02C6"/>
        </inputs>
        <outputs>
            <dataPoint pairingId="0030" dpt="0501" matchCode="30000000" maxConnections="10" autoConnectHouse="false" autoConnectFloor="false" autoConnectRoom="true" defaultConnection="false" nameId="011B"/>
            <dataPoint pairingId="0032" dpt="0501" matchCode="30000000" maxConnections="10" autoConnectHouse="false" autoConnectFloor="false" autoConnectRoom="true" defaultConnection="false" nameId="0021"/>
            <dataPoint pairingId="0033" dpt="0901" matchCode="00000003" maxConnections="10" autoConnectHouse="false" autoConnectFloor="false" autoConnectRoom="false" defaultConnection="true" nameId="0022"/>
            <dataPoint pairingId="0034" dpt="0901" matchCode="00000003" maxConnections="10" autoConnectHouse="false" autoConnectFloor="false" autoConnectRoom="false" defaultConnection="true" nameId="010F"/>
            <dataPoint pairingId="0038" dpt="0101" matchCode="00000003" maxConnections="10" autoConnectHouse="false" autoConnectFloor="false" autoConnectRoom="false" defaultConnection="true" nameId="0112"/>
            <dataPoint pairingId="0130" dpt="0901" matchCode="00000003" maxConnections="10" autoConnectHouse="false" autoConnectFloor="false" autoConnectRoom="false" defaultConnection="true" nameId="0113"/>
            <dataPoint pairingId="0036" dpt="1501" matchCode="00000003" maxConnections="10" autoConnectHouse="false" autoConnectFloor="false" autoConnectRoom="false" defaultConnection="true" nameId="0025"/>
            <dataPoint pairingId="014D" dpt="0501" matchCode="30000000" maxConnections="10" autoConnectHouse="false" autoConnectFloor="false" autoConnectRoom="false" defaultConnection="false" nameId="039B"/>
            <dataPoint pairingId="014E" dpt="0501" matchCode="30000000" maxConnections="10" autoConnectHouse="false" autoConnectFloor="false" autoConnectRoom="false" defaultConnection="false" nameId="039C"/>
            <dataPoint pairingId="0136" dpt="050A" matchCode="10000000" maxConnections="10" autoConnectHouse="false" autoConnectFloor="false" autoConnectRoom="true" defaultConnection="false" nameId="0331"/>
            <dataPoint pairingId="0147" dpt="050A" matchCode="10000000" maxConnections="10" autoConnectHouse="false" autoConnectFloor="false" autoConnectRoom="true" defaultConnection="false" nameId="02C3"/>
            <dataPoint pairingId="0031" dpt="1465" matchCode="00000003" maxConnections="10" autoConnectHouse="false" autoConnectFloor="false" autoConnectRoom="false" defaultConnection="true" nameId="0110"/>
            <dataPoint pairingId="0037" dpt="0101" matchCode="00000003" maxConnections="10" autoConnectHouse="false" autoConnectFloor="false" autoConnectRoom="false" defaultConnection="true" nameId="0118"/>
            <dataPoint pairingId="014B" dpt="0101" matchCode="00000003" maxConnections="10" autoConnectHouse="false" autoConnectFloor="false" autoConnectRoom="false" defaultConnection="false" nameId="0335"/>
            <dataPoint pairingId="014C" dpt="0101" matchCode="00000003" maxConnections="10" autoConnectHouse="false" autoConnectFloor="false" autoConnectRoom="false" defaultConnection="false" nameId="0336"/>
        </outputs>
        <parameters>
            <parameter matchCode="30000003" channelSelector="false" dpt="0901" nameId="011D" parameterId="0018" accessLevel="Enduser" visible="false" writable="true" sync="none">
                <valueFloat factor="1.0000" stepWidth="0.5000" maxValue="35.0000" minValue="15" defaultValue="21.0000"/>
            </parameter>
            <parameter matchCode="30000003" dpt="0901" nameId="011E" parameterId="0019" accessLevel="Enduser" visible="true" writable="true">
                <valueFloat name="Eco&#x20;Delta&#x20;Temperature" factor="1" stepWidth="0.5" maxValue="5" minValue="1" defaultValue="3"/>
            </parameter>
            <parameter matchCode="30000003" channelSelector="false" dpt="0901" nameId="0207" parameterId="001D" accessLevel="Enduser" visible="false" writable="false" sync="none">
                <valueFloat factor="1.0000" stepWidth="0.5000" maxValue="14.0000" minValue="14.0000" defaultValue="14.0000"/>
            </parameter>
            <parameter matchCode="30000003" channelSelector="false" dpt="0901" nameId="0208" parameterId="001E" accessLevel="Enduser" visible="false" writable="false" sync="none">
                <valueFloat factor="1.0000" stepWidth="0.5000" maxValue="-14.0000" minValue="-14.0000" defaultValue="-14.0000"/>
            </parameter>
            <parameter matchCode="10000001" dpt="0102" nameId="01BB" parameterId="0020" accessLevel="Enduser" visible="false" writable="false" sync="source">
                <valueUnsigned factor="1.0" stepWidth="1" maxValue="1" minValue="0" defaultValue="1"/>
            </parameter>
            <parameter matchCode="20000002" dpt="0102" nameId="01BB" parameterId="0020" accessLevel="Enduser" visible="false" writable="false" sync="source">
                <valueUnsigned factor="1.0" stepWidth="1" maxValue="1" minValue="0" defaultValue="0"/>
            </parameter>
        </parameters>
        <scenes>
            <scene>
                <storedDataPoint pairingId="0033"/>
                <storedDataPoint pairingId="0036"/>
                <storedDataPoint pairingId="0038"/>
            </scene>
            <scene>
                <storedDataPoint pairingId="0033"/>
                <storedDataPoint pairingId="0036"/>
                <storedDataPoint pairingId="0038"/>
            </scene>
            <scene>
                <storedDataPoint pairingId="0033"/>
                <storedDataPoint pairingId="0036"/>
                <storedDataPoint pairingId="0038"/>
            </scene>
            <scene>
                <storedDataPoint pairingId="0033"/>
                <storedDataPoint pairingId="0036"/>
                <storedDataPoint pairingId="0038"/>
            </scene>
            <scene>
                <storedDataPoint pairingId="0033"/>
                <storedDataPoint pairingId="0036"/>
                <storedDataPoint pairingId="0038"/>
            </scene>
            <scene>
                <storedDataPoint pairingId="0033"/>
                <storedDataPoint pairingId="0036"/>
                <storedDataPoint pairingId="0038"/>
            </scene>
            <scene>
                <storedDataPoint pairingId="0033"/>
                <storedDataPoint pairingId="0036"/>
                <storedDataPoint pairingId="0038"/>
            </scene>
            <scene>
                <storedDataPoint pairingId="0033"/>
                <storedDataPoint pairingId="0036"/>
                <storedDataPoint pairingId="0038"/>
            </scene>
        </scenes>
    </channel>
</descriptions>

The problem is that I couldn't find any way to associate each datapoint to its pairingId. Maybe you can verify what XML you are getting. In the mean time I'm gonna use the last version (with manually edited datapoints, since mine are different from yours).

@EnricoBilla Thanks for the feedback, although it is... unfortunate ;-)

The heart of this PR is that we now load a much more enhanced configuration via getAll RPC call. This enhanced version includes pairingIds and many more information directly in the channel (see the fixture files that I extracted from my config). It looks like your XML is much more reduced, just like before.

Could you try to play with the first numerical parameter for the getAll RPC call here and see if you can retrieve the "enhanced" XML file? I achieved it by changing the first numerical paramter from '4' to '2'. The second parameter controls pretty-printing the XML file, and the third parameter - I don't know. Or try it here in the "live" file and see if you can get this component running.

@minze If I read your config correctly, you seem to be running a Busch-Jaeger SysAP with sw version 2.5.5, right? If so, it may have something to do with the ABB variant.

From my experience the datapoint numbers seem to be ascending from 0, so in your example datapoint idp0000 should be pairing ID 0039, datapoint idp0001 should be pairing ID 0042 and so on. I think we should not rely on this, and a "hard" association would be better, but maybe we can use this to associate datapoints and pairing IDs if everything else fails.

@Tho85 Well, you are right, I changed the '2' in getAll to a '4' because I had other errors, so I tried to change it back to where it was before; it solved that problem but caused another to show up.

Now moving that '4' back to '2' it's causing again the same issue as before (but devices appears to have pairingIds in them, so your integration should work fine):

2020-10-28 00:42:26 INFO (MainThread) [custom_components.freeathome.fah.pfreeathome] send presence
2020-10-28 00:42:26 INFO (MainThread) [custom_components.freeathome.fah.pfreeathome] get roster
2020-10-28 00:42:28 ERROR (MainThread) [homeassistant.config_entries] Error setting up entry 192.168.1.X for freeathome
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 231, in async_setup
    result = await component.async_setup_entry(hass, self)  # type: ignore
  File "/config/custom_components/freeathome/__init__.py", line 75, in async_setup_entry
    await sysap.find_devices()
  File "/config/custom_components/freeathome/fah/pfreeathome.py", line 774, in find_devices
    await self.xmpp.find_devices(self._use_room_names)
  File "/config/custom_components/freeathome/fah/pfreeathome.py", line 559, in find_devices
    root = ET.fromstring(config)
  File "/usr/local/lib/python3.8/xml/etree/ElementTree.py", line 1320, in XML
    parser.feed(text)
  File "<string>", line None
xml.etree.ElementTree.ParseError: duplicate attribute: line 1, column 241090

I have to troubleshoot it a little bit more. Thanks a lot for the hint :)

xml.etree.ElementTree.ParseError: duplicate attribute: line 1, column 241090

Hmm, this looks like it is an invalid XML file. The error comes from Python's internal XML parser, so I don't know what's going on there. Maybe change the second numeric parameter to '1' to get a readable XML file (with line breaks) and look if there's something odd in there?

Well, I went to that column and here is what I did found (sorry for the ver very long code):

<device isTp="true" domainAddress="BAE7" individualAddress="0146" nameId="00EA" profile="0E00" maxAPDULength="39" compilerVersion="00000000" buildNumber="00000000" iconId="FFF6" protocolVersion="0000" minConfigVersion="0000" deviceFlavor="00" interface="TP" functionId="FEFE" shortSerialNumber="MYN" softwareId="0853" softwareVersion="1.83" deviceId="010D" name="2CSYE1202" serialNumber="ABB5000XXXXX" commissioningState="ready" consistencyTag="0" copyId="1" progress="100">
    <channels>
        <channel channelId="0805" name="Room-Temperature-Controller" maxScenes="08" mask="FFFFFFFF" nameId="0062" i="ch0000" cid="ABB0ABCD">
            <attribute name="displayName">Thermostat</attribute>
            <attribute name="floor">02</attribute>
            <attribute name="functionId">23</attribute>
            <attribute name="offsetX">0.593206592228102</attribute>
            <attribute name="offsetY">0.600826188126858</attribute>
            <attribute name="room">09</attribute>
            <attribute name="selectedIcon">21</attribute>
            <functions>
                <function functionId="000A" nameId="01BB" sensorMatchCode="10000000" actuatorMatchCode="00000001" accessLevel="Enduser" isDefault="false" fixed="false" bestMatch="true" name="RTC&#x20;has&#x20;connected&#x20;the&#x20;fan&#x20;coil"/>
                <function functionId="0023" nameId="01BA" sensorMatchCode="20000000" actuatorMatchCode="00000002" accessLevel="Enduser" isDefault="true" fixed="false" bestMatch="true" name="Room&#x20;temperature&#x20;controller"/>
            </functions>
            <inputs>
                <dataPoint defaultConnection="false" name="Relative&#x20;Set&#x20;Point&#x20;Request" nameId="0115" matchCode="00000003" maxConnections="10" pairingId="0039" dpt="0901" autoConnectRoom="false" autoConnectFloor="false" autoConnectHouse="false" name="Relative&#x20;Set&#x20;Point&#x20;Request" full="false" i="idp0000">
                    <value>invalid</value>
                </dataPoint>
                <dataPoint defaultConnection="false" name="Controller&#x20;On/Off&#x20;Request" nameId="0119" matchCode="00000003" maxConnections="10" pairingId="0042" dpt="0101" autoConnectRoom="false" autoConnectFloor="false" autoConnectHouse="false" name="Controller&#x20;On/Off&#x20;Request" full="false" i="idp0001">
                    <value>1</value>
                </dataPoint>
                <dataPoint defaultConnection="false" name="Eco&#x20;mode&#x20;On/Off&#x20;Request" nameId="0117" matchCode="00000003" maxConnections="10" pairingId="003A" dpt="0101" autoConnectRoom="false" autoConnectFloor="false" autoConnectHouse="false" name="Eco&#x20;mode&#x20;On/Off&#x20;Request" full="false" i="idp0002">
                    <value>0</value>
                </dataPoint>
                <dataPoint defaultConnection="false" name="Scene&#x20;Control" nameId="000E" matchCode="00000003" maxConnections="20" pairingId="0004" dpt="1201" autoConnectRoom="false" autoConnectFloor="false" autoConnectHouse="false" name="Scene&#x20;Control" full="false" i="idp0003">
                    <value>0</value>
                </dataPoint>
                <dataPoint defaultConnection="false" name="Absolute&#x20;set-point&#x20;temperature" nameId="0205" matchCode="00000003" maxConnections="01" pairingOptional="false" pairingId="0140" dpt="0901" autoConnectRoom="false" autoConnectFloor="false" autoConnectHouse="false" name="Absolute&#x20;setpoint&#x20;temperature" full="false" i="idp0004">
                    <value>invalid</value>
                </dataPoint>
                <dataPoint defaultConnection="false" name="Info&#x20;value&#x20;heating" nameId="017D" matchCode="30000000" maxConnections="10" pairingId="0131" dpt="0501" autoConnectRoom="false" autoConnectFloor="false" autoConnectHouse="false" name="Info&#x20;Value&#x20;Heating" full="false" i="idp0005">
                    <value>0</value>
                    <address timeSlot="0" primary="true" singleSink="false">a15d</address>
                </dataPoint>
                <dataPoint defaultConnection="false" name="Info&#x20;value&#x20;cooling" nameId="01C7" matchCode="30000000" maxConnections="10" pairingId="0132" dpt="0501" autoConnectRoom="false" autoConnectFloor="false" autoConnectHouse="false" name="Info&#x20;value&#x20;cooling" full="false" i="idp0006">
                    <value>0</value>
                </dataPoint>
                <dataPoint defaultConnection="false" name="Heating&#x20;Demand&#x20;Info" nameId="039E" matchCode="00000003" maxConnections="01" pairingId="014F" dpt="0501" autoConnectRoom="false" autoConnectFloor="false" autoConnectHouse="false" name="Heating&#x20;demand&#x20;feedback&#x20;signal" full="false" i="idp0007">
                    <value>0</value>
                </dataPoint>
                <dataPoint defaultConnection="false" name="Cooling&#x20;Demand&#x20;Info" nameId="039F" matchCode="00000003" maxConnections="01" pairingId="0150" dpt="0501" autoConnectRoom="false" autoConnectFloor="false" autoConnectHouse="false" name="Cooling&#x20;demand&#x20;feedback&#x20;signal" full="false" i="idp0008">
                    <value>0</value>
                </dataPoint>
                <dataPoint defaultConnection="false" name="Window/Door" nameId="012E" matchCode="00000003" maxConnections="10" pairingId="0035" dpt="0113" autoConnectRoom="false" autoConnectFloor="false" autoConnectHouse="false" name="Window/Door" full="false" i="idp0009">
                    <value>0</value>
                </dataPoint>
                <dataPoint defaultConnection="false" name="Fan&#x20;Stage&#x20;Request" nameId="0116" matchCode="00000003" maxConnections="10" pairingId="0040" dpt="1465" autoConnectRoom="false" autoConnectFloor="false" autoConnectHouse="false" name="Fan&#x20;Level&#x20;Request" full="false" i="idp000A">
                    <value>0</value>
                </dataPoint>
                <dataPoint defaultConnection="false" name="Info&#x20;Fan&#x20;Actuating&#x20;Stage&#x20;Heating" nameId="01F0" matchCode="10000000" maxConnections="10" pairingId="0145" dpt="050A" autoConnectRoom="true" autoConnectFloor="false" autoConnectHouse="false" name="Info&#x20;Actuating&#x20;Fan&#x20;Stage&#x20;Heating" full="false" i="idp000B">
                    <value>0</value>
                </dataPoint>
                <dataPoint defaultConnection="false" name="Info&#x20;Fan&#x20;Manual&#x20;On&#x20;Off&#x20;Heating" nameId="01F1" matchCode="10000000" maxConnections="10" pairingId="0146" dpt="0101" autoConnectRoom="true" autoConnectFloor="false" autoConnectHouse="false" name="Info&#x20;Actuating&#x20;Fan&#x20;Manual&#x20;On/Off&#x20;Heating" full="false" i="idp000C">
                    <value>0</value>
                </dataPoint>
                <dataPoint defaultConnection="false" name="Info&#x20;Fan&#x20;Actuating&#x20;Stage&#x20;Cooling" nameId="02C5" matchCode="10000000" maxConnections="10" pairingId="0149" dpt="050A" autoConnectRoom="true" autoConnectFloor="false" autoConnectHouse="false" name="Info&#x20;Fan&#x20;Stage&#x20;Cooling" full="false" i="idp000D">
                    <value>0</value>
                </dataPoint>
                <dataPoint defaultConnection="false" name="Info&#x20;Fan&#x20;Manual&#x20;On&#x20;Off&#x20;Cooling" nameId="02C6" matchCode="10000000" maxConnections="10" pairingId="014A" dpt="0101" autoConnectRoom="true" autoConnectFloor="false" autoConnectHouse="false" name="Info&#x20;Fan&#x20;Manual&#x20;On/Off&#x20;Cooling" full="false" i="idp000E">
                    <value>0</value>
                </dataPoint>
            </inputs>
            <outputs>
                <dataPoint defaultConnection="false" name="Actuating&#x20;Value&#x20;Heating" nameId="011B" matchCode="30000000" maxConnections="10" pairingId="0030" dpt="0501" autoConnectRoom="true" autoConnectFloor="false" autoConnectHouse="false" name="Actuating&#x20;Value&#x20;Heating" full="false" i="odp0000">
                    <value>0</value>
                    <address timeSlot="0" primary="true" singleSink="false">5339</address>
                </dataPoint>
                <dataPoint defaultConnection="false" name="Actuating&#x20;Value&#x20;Cooling" nameId="0021" matchCode="30000000" maxConnections="10" pairingId="0032" dpt="0501" autoConnectRoom="true" autoConnectFloor="false" autoConnectHouse="false" name="Actuating&#x20;Value&#x20;Cooling" full="false" i="odp0001">
                    <value>0</value>
                </dataPoint>
                <dataPoint defaultConnection="true" name="Set&#x20;Value&#x20;Temperature" nameId="0022" matchCode="00000003" maxConnections="10" pairingId="0033" dpt="0901" autoConnectRoom="false" autoConnectFloor="false" autoConnectHouse="false" name="Set&#x20;Value&#x20;Temperature" full="false" i="odp0002">
                    <value>15</value>
                    <address timeSlot="0" primary="true" singleSink="false">6839</address>
                </dataPoint>
                <dataPoint defaultConnection="true" name="Relative&#x20;Set&#x20;Point&#x20;Temperature" nameId="010F" matchCode="00000003" maxConnections="10" pairingId="0034" dpt="0901" autoConnectRoom="false" autoConnectFloor="false" autoConnectHouse="false" name="Relative&#x20;Set&#x20;Point&#x20;Temperature" full="false" i="odp0003">
                    <value>-6</value>
                    <address timeSlot="0" primary="true" singleSink="false">475c</address>
                </dataPoint>
                <dataPoint defaultConnection="true" name="Controller&#x20;On/Off" nameId="0112" matchCode="00000003" maxConnections="10" pairingId="0038" dpt="0101" autoConnectRoom="false" autoConnectFloor="false" autoConnectHouse="false" name="Controller&#x20;On/Off" full="false" i="odp0004">
                    <value>1</value>
                    <address timeSlot="0" primary="true" singleSink="false">6b3c</address>
                </dataPoint>
                <dataPoint defaultConnection="true" name="Measured&#x20;Temperature" nameId="0113" matchCode="00000003" maxConnections="10" pairingId="0130" dpt="0901" autoConnectRoom="false" autoConnectFloor="false" autoConnectHouse="false" name="Measured&#x20;Temperature" full="false" i="odp0005">
                    <value>18.7</value>
                    <address timeSlot="0" primary="true" singleSink="false">6981</address>
                </dataPoint>
                <dataPoint defaultConnection="true" name="Status&#x20;indication" nameId="0025" matchCode="00000003" maxConnections="10" pairingId="0036" dpt="1501" autoConnectRoom="false" autoConnectFloor="false" autoConnectHouse="false" name="Status&#x20;indication" full="false" i="odp0006">
                    <value>97</value>
                    <address timeSlot="0" primary="true" singleSink="false">68b6</address>
                </dataPoint>
                <dataPoint defaultConnection="false" name="Heating&#x20;demand" nameId="039B" matchCode="30000000" maxConnections="10" pairingId="014D" dpt="0501" autoConnectRoom="false" autoConnectFloor="false" autoConnectHouse="false" name="Heating&#x20;demand" full="false" i="odp0007">
                    <value>0</value>
                </dataPoint>
                <dataPoint defaultConnection="false" name="Cooling&#x20;demand" nameId="039C" matchCode="30000000" maxConnections="10" pairingId="014E" dpt="0501" autoConnectRoom="false" autoConnectFloor="false" autoConnectHouse="false" name="Cooling&#x20;demand" full="false" i="odp0008">
                    <value>0</value>
                </dataPoint>
                <dataPoint defaultConnection="false" name="Actuating&#x20;Fan&#x20;Stage&#x20;Heating" nameId="0331" matchCode="10000000" maxConnections="10" pairingId="0136" dpt="050A" autoConnectRoom="true" autoConnectFloor="false" autoConnectHouse="false" name="Actuating&#x20;Fan&#x20;Stage&#x20;Heating" full="false" i="odp0009">
                    <value>0</value>
                </dataPoint>
                <dataPoint defaultConnection="false" name="Actuating&#x20;Fan&#x20;Stage&#x20;Cooling" nameId="02C3" matchCode="10000000" maxConnections="10" pairingId="0147" dpt="050A" autoConnectRoom="true" autoConnectFloor="false" autoConnectHouse="false" name="Actuating&#x20;Fan&#x20;Stage&#x20;Cooling" full="false" i="odp000A">
                    <value>0</value>
                </dataPoint>
                <dataPoint defaultConnection="true" name="Fan&#x20;Coil&#x20;Level" nameId="0110" matchCode="00000003" maxConnections="10" pairingId="0031" dpt="1465" autoConnectRoom="false" autoConnectFloor="false" autoConnectHouse="false" name="Fan&#x20;Level&#x20;Heating" full="false" i="odp000B">
                    <value>0</value>
                    <address timeSlot="0" primary="true" singleSink="false">265c</address>
                </dataPoint>
                <dataPoint defaultConnection="true" name="Fan&#x20;Manual&#x20;On&#x20;Off" nameId="0118" matchCode="00000003" maxConnections="10" pairingId="0037" dpt="0101" autoConnectRoom="false" autoConnectFloor="false" autoConnectHouse="false" name="Fan&#x20;Manual&#x20;On/Off&#x20;Request" full="false" i="odp000C">
                    <value>0</value>
                    <address timeSlot="0" primary="true" singleSink="false">692e</address>
                </dataPoint>
                <dataPoint defaultConnection="false" name="AL_HEATING_ACTIVE" nameId="0335" matchCode="00000003" maxConnections="10" pairingId="014B" dpt="0101" autoConnectRoom="false" autoConnectFloor="false" autoConnectHouse="false" name="Heating&#x20;active" full="false" i="odp000D">
                    <value>0</value>
                </dataPoint>
                <dataPoint defaultConnection="false" name="AL_COOLING_ACTIVE" nameId="0336" matchCode="00000003" maxConnections="10" pairingId="014C" dpt="0101" autoConnectRoom="false" autoConnectFloor="false" autoConnectHouse="false" name="Cooling&#x20;active" full="false" i="odp000E">
                    <value>0</value>
                </dataPoint>
            </outputs>
            [...]
        </channel>
    </channels>
    <dicts/>
</device>

It appears that some dataPoints contains the attribute 'name' multiple times, which is against XML specification. I don't know why tho...

Thanks for the XML, this will make a good fixture for writing a test.

It's interesting that the SysAP returns wrong XML. I think the reason may be hidden deep within their proprietary software... I can't think of a clean solution right now, maybe I'll strip the names from datapoints altogether via some ugly regex...

That's ugly, but it'll work I guess... I can't think of a better solution; maybe I'll try tomorrow. Thanks for the quick responses!

I've added the ugliness, could you try again?

I tried it, it didn't work because the regex matched the symbols < > between the attributes. Editing it to re.sub(r'name="[^"]*" ([^>]*)name="[^"]*"', r'\1', xml) works.

At least it starts without errors, now I have the problem that covers are not available; but I'll check it out later. Everything else is working, thanks!

@Minze If I read your config correctly, you seem to be running a Busch-Jaeger SysAP with sw version 2.5.5, right? If so, it may have something to do with the ABB variant.

@miezie

I run the latest sysap version from Busch indeed.

Regards, Minze

I tried it, it didn't work because the regex matched the symbols < > between the attributes. Editing it to re.sub(r'name="[^"]*" ([^>]*)name="[^"]*"', r'\1', xml) works.

... and this is why I shouldn't be coding late at night. Thanks for the regex, I'll change the tests and code later. At night, I guess ;-)

At least it starts without errors, now I have the problem that covers are not available; but I'll check it out later. Everything else is working, thanks!

Which kind of cover actuator do you have, and in which configuration? Could you paste the device xml?

... and this is why I shouldn't be coding late at night. Thanks for the regex, I'll change the tests and code later. At night, I guess ;-)

I would've committed it, but I would've had to open a PR on the PR :)

Which kind of cover actuator do you have, and in which configuration? Could you paste the device xml?

Here is the XML of the cover device:

<device isTp="true" domainAddress="BAE7" individualAddress="013A" nameId="00E5" profile="0E00" maxAPDULength="39" compilerVersion="00000000" buildNumber="00000000" iconId="FFF9" protocolVersion="0000" minConfigVersion="0000" deviceFlavor="00" interface="TP" functionId="FEFB" shortSerialNumber="PEA" softwareId="0833" softwareVersion="1.51" deviceId="0109" name="2CSYE1105" serialNumber="ABB5000ABCDE" commissioningState="ready" consistencyTag="5" copyId="2" progress="100">
    <channels>
        <channel channelId="0502" name="Shutter&#x20;Actuator" maxScenes="08" mask="FFFFFFFF" nameId="00E4" i="ch0000" cid="ABB50502">
            <attribute name="displayName">Cover</attribute>
            <attribute name="floor">02</attribute>
            <attribute name="functionId">2c</attribute>
            <attribute name="offsetX">0.9256450670442017</attribute>
            <attribute name="offsetY">0.5492448806762695</attribute>
            <attribute name="room">0A</attribute>
            <attribute name="selectedIcon">3</attribute>
            <functions>
                <function functionId="002C" nameId="00E4" sensorMatchCode="00000000" actuatorMatchCode="00000001" isDefault="true" />
            </functions>
            <inputs>
                <dataPoint nameId="0015" matchCode="00000001" maxConnections="10" pairingId="0020" dpt="0108" full="false" i="idp0000">
                    <value>0</value>
                </dataPoint>
                <dataPoint nameId="0016" matchCode="00000001" maxConnections="10" pairingId="0021" dpt="0107" full="false" i="idp0001">
                    <value>0</value>
                </dataPoint>
                <dataPoint nameId="000E" matchCode="00000001" maxConnections="20" pairingId="0004" dpt="1201" full="false" i="idp0002">
                    <value>0</value>
                </dataPoint>
            </inputs>
            <outputs>
                <dataPoint defaultConnection="true" nameId="001D" matchCode="00000001" maxConnections="1" pairingId="0120" dpt="1465" full="false" i="odp0000">
                    <value>0</value>
                    <address timeSlot="0" primary="true" singleSink="false">e70b</address>
                </dataPoint>
            </outputs>
            <parameters>
                <parameter nameId="0028" parameterId="FFFF" accessLevel="Enduser" matchCode="00000001" dpt="1464" visible="true" writable="true" i="pm0000" optional="false">
                    <valueEnum>
                        <option key="0" nameId="023C" isDefault="true" />
                        <option key="1" nameId="0029" />
                    </valueEnum>
                    <value>0</value>
                </parameter>
                <parameter nameId="002C" parameterId="FFFF" matchCode="00000001" dpt="1464" visible="true" writable="true" i="pm0001" optional="false">
                    <valueEnum>
                        <option key="2" nameId="002E" isDefault="true" />
                        <option key="3" nameId="00F5" />
                        <option key="0" nameId="00F6" />
                        <option key="1" nameId="00F7" />
                    </valueEnum>
                    <value>0</value>
                </parameter>
            </parameters>
            <scenes>
                <scene active="false" sceneNumber="0" i="sc0000">
                    <storedDataPoint pairingId="120">
                        <value>64</value>
                    </storedDataPoint>
                </scene>
                [...]
            </scenes>
        </channel>
    </channels>
    <dicts/>
</device>

It seems to have a function id of 0x002C but it doesn't appear on the official documentation... Later I'll try to add 0x002C to the cover ids to see whether it works

@jheling Nice! :tada:

@EnricoBilla Do you want to submit a second pull request for the regex, and maybe the cover as well?

@Tho85 Ok, I can do it. But first I need to check if adding 002C to the cover's function ids is enough.

jheling / freeathome