Issue with Fritz OS 7.70beta

[JSoko]

https://github.com/home-assistant/core/issues/102044

[kbr]

According to the given error-log https://github.com/home-assistant/core/files/12909570/home-assistant_fritz_2023-10-15T12-43-28.790Z.log (mentioned in https://github.com/home-assistant/core/issues/102044) the xml-data returned from the router are incomplete. Therefore the error.

To check whether this is an invalid xml-response from the router you can access the xml-data manually by means of a browser: http://192.168.178.1:49000/tr64desc.xml

According to AVM the Fritz!OS 7.70 beta is a prerelease especially for the 5590 Fiber, 5530 Fiber, 7590 AX and 7530 AX models. Unfortunately I don't have any of these routers. It would be helpful if you can provide the mentioned tr64desc.xml file.

[JSoko]

Thank You for Your fast response!

I'll also inform AVM...

Attached you find the xml. tr64desc.zip

[kbr]

Thank you very much for providing the xml-file. The response from the router is ok. I've tested the parsing-engine of fritzconnection with the tr64desc.xml response from the 5590 router as input and this works without any problems.

So currently it is unclear why fritzconnection gets an invalid (or incomplete) xml-input in combination with the HA integration. According to the HA-error log it seems to be that the last line of the tr64desc input has been suppressed.

Next step could be to try to reproduce this with a plain fritzconnection instance on a 5590 router:

>>> from fritzconnection import FritzConnection
>>> # provide user and password if not stored in the environment:
>>> fc = FritzConnection(address="192.168.178.1", user="username", password="the_password") 
>>> fc

In case of success this should report the router model like (here my 7590):

FRITZ!Box 7590 at http://192.168.178.1/
FRITZ!OS: 7.56

If this behaves the same as in HA this should result in an xml-parsing error.

[JSoko]

That's what I get on 5590 7.70beta:

>>> fc = FritzConnection(address="192.168.1.1", user="Home_Assistant", password="***")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\micro\AppData\Local\Programs\Python\Python312\Lib\site-packages\fritzconnection\core\fritzconnection.py", line 285, in __init__
    self._load_router_api(
  File "C:\Users\micro\AppData\Local\Programs\Python\Python312\Lib\site-packages\fritzconnection\core\fritzconnection.py", line 561, in _load_router_api
    self._load_api_from_router()
  File "C:\Users\micro\AppData\Local\Programs\Python\Python312\Lib\site-packages\fritzconnection\core\fritzconnection.py", line 657, in _load_api_from_router
    self.device_manager.add_description(source)
  File "C:\Users\micro\AppData\Local\Programs\Python\Python312\Lib\site-packages\fritzconnection\core\devices.py", line 76, in add_description
    root = get_xml_root(source, timeout=self.timeout, session=self.session)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\micro\AppData\Local\Programs\Python\Python312\Lib\site-packages\fritzconnection\core\utils.py", line 83, in get_xml_root
    return etree.fromstring(source)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\micro\AppData\Local\Programs\Python\Python312\Lib\xml\etree\ElementTree.py", line 1324, in XML
    return parser.close()
           ^^^^^^^^^^^^^^
xml.etree.ElementTree.ParseError: no element found: line 292, column 0

If I do it to a 7590 in my network, everything wents OK:

>>> fc = FritzConnection(address="192.168.1.8", user="Home_Assistant", password="***")
>>> fc
FRITZ!Box 7590 at http://192.168.1.8
FRITZ!OS: 7.57

[desosav]

This is what i get on 5530 (7.70 beta):

>>> fc = FritzConnection(address="192.168.178.1", user="user", password="****")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\testuser\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\fritzconnection\core\fritzconnection.py", line 285, in __init__
    self._load_router_api(
  File "C:\Users\testuser\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\fritzconnection\core\fritzconnection.py", line 561, in _load_router_api
    self._load_api_from_router()
  File "C:\Users\testuser\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\fritzconnection\core\fritzconnection.py", line 657, in _load_api_from_router
    self.device_manager.add_description(source)
  File "C:\Users\testuser\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\fritzconnection\core\devices.py", line 76, in add_description
    root = get_xml_root(source, timeout=self.timeout, session=self.session)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\testuser\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\fritzconnection\core\utils.py", line 83, in get_xml_root
    return etree.fromstring(source)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1776.0_x64__qbz5n2kfra8p0\Lib\xml\etree\ElementTree.py", line 1339, in XML
    return parser.close()
           ^^^^^^^^^^^^^^
xml.etree.ElementTree.ParseError: no element found: line 257, column 0

[kbr]

@JSoko , @desosav thank you very much for the response. It's hard to reproduce this without the proper router and beta software. It seems to be that the router returns a proper xml-file from the url "http://192.168.178.1:49000/tr64desc.xml" by means of a browser but returns something different if this request is made by means of the requests library. This seems to be unlikely, but may be so far the best explanation for observed behaviour.

If fritzconnection is installed then requests is also installed and the following code should work (adapt the IP if necessary):

import requests

url = "http://192.168.178.1:49000/tr64desc.xml"

res = requests.get(url, verify=False)
print(res.encoding)
print(res.content[-140:])
print()
print(res.text[-140:])

This should return something like (again from my 7590):

ISO-8859-1
b'e>\n</serviceList>\n</device>\n</deviceList>\n</device>\n</deviceList>\n<presentationURL>http://192.168.178.1</presentationURL>\n</device>\n</root>\n'

e>
</serviceList>
</device>
</deviceList>
</device>
</deviceList>
<presentationURL>http://192.168.178.1</presentationURL>
</device>
</root>

This output can be considered as complete and with some guessed encoding which is unproblematic as long as we are in low-ascii. (The xml-response from the router provides no encoding information.)

The next test reports what is seen by library as a response from the router (in the file "xml_traffic.log")

import logging

from fritzconnection.core.utils import get_content_from
from fritzconnection.core.logger import activate_local_debug_mode
from fritzconnection.core.logger import reset

activate_local_debug_mode(handler=logging.FileHandler("xml_traffic.log"))
url = "http://192.168.178.1:49000/tr64desc.xml"
content = get_content_from(url)
print(content[-150:])

reset()

If you can provide these contents, may be we are able to get an idea where to search for the reason of this issue.

[JSoko]

1st output 5590:

>>> import requests
>>> url = "http://192.168.1.1:49000/tr64desc.xml"
>>>
>>> res = requests.get(url, verify=False)
>>> print(res.encoding)
ISO-8859-1
>>> print(res.content[-140:])
b'/upnp/control/x_wanfiber</controlURL>\n<eventSubURL>/upnp/control/x_wanfiber</eventSubURL>\n<SCPDURL>/x_wanfiberSCPD.xml</SCPDURL>\n</service>\n'
>>> print()

>>> print(res.text[-140:])
/upnp/control/x_wanfiber</controlURL>
<eventSubURL>/upnp/control/x_wanfiber</eventSubURL>
<SCPDURL>/x_wanfiberSCPD.xml</SCPDURL>
</service>

2nd output

>>> import logging
>>> from fritzconnection.core.utils import get_content_from
>>> from fritzconnection.core.logger import activate_local_debug_mode
>>> from fritzconnection.core.logger import reset
>>> activate_local_debug_mode(handler=logging.FileHandler("xml_traffic.log"))
>>> url = "http://192.168.1.1:49000/tr64desc.xml"
>>> content = get_content_from(url)
>>> print(content[-150:])
ontrolURL>/upnp/control/x_wanfiber</controlURL>
<eventSubURL>/upnp/control/x_wanfiber</eventSubURL>
<SCPDURL>/x_wanfiberSCPD.xml</SCPDURL>
</service>

>>> reset()
>>>

[desosav]

i've got the exact same outputs reported by JSoko

[kbr]

Thank you – unbelievable but true: the requests library does not return the complete response (this library is rather popular and well tested). The response must end with </root>. I suppose the output in xml_traffic.log also does not include the last line compared to the result by accessing the url by means of a browser.

If you have access to the curl tool you may also test another way to access the router:

curl http://192.168.178.1:49000/tr64desc.xml

The output should also end with </root> (in my case it does). If you can provide this output "as is", may be we can see whether there is something different from a normal text-response that may confuse requests.

I suppose we have to get in contact with AVM to handle this. But for that we should have some beef.

[JSoko]

Thank you very much for your efforts.

This morning I reported it as a bug with reference to this issue on github to AVM using the beta reporting function. I don't know whether Avm takes this into account. I didn't receive a response.

They will probably be more responsive to you as a developer?!?

@desosav Can you perhaps also report the error to avm using the feedback form from your 5530?

curl on a 7590 ends with root for me

As you expected, curl on the 5590 with 7.70 does not end with root, but as follows:

C:\Users\micro>curl http://192.168.1.1:49000/tr64desc.xml
<?xml version="1.0"?>
<root xmlns="urn:dslforum-org:device-1-0">
<specVersion>
<major>1</major>
<minor>0</minor>
</specVersion>
<systemVersion>
<HW>272</HW>
<Major>272</Major>
<Minor>7</Minor>
<Patch>70</Patch>
<Buildnumber>108606</Buildnumber>
<Display>272.07.70-108606</Display>
</systemVersion>
<device>
<deviceType>urn:dslforum-org:device:InternetGatewayDevice:1</deviceType>
<friendlyName>001-5590ez</friendlyName>
<manufacturer>AVM</manufacturer>
<manufacturerURL>www.avm.de</manufacturerURL>
<modelDescription>FRITZ!Box 5590 Fiber</modelDescription>
<modelName>FRITZ!Box 5590 Fiber</modelName>
<modelNumber>5590 - avme</modelNumber>
<modelURL>www.avm.de</modelURL>

[...]

<deviceType>urn:dslforum-org:device:WANConnectionDevice:1</deviceType>
<friendlyName>WANConnectionDevice - FRITZ!Box 5590 Fiber</friendlyName>
<manufacturer>AVM</manufacturer>
<manufacturerURL>www.avm.de</manufacturerURL>
<modelDescription>WANConnectionDevice - FRITZ!Box 5590 Fiber</modelDescription>
<modelName>WANConnectionDevice - FRITZ!Box 5590 Fiber</modelName>
<modelNumber>5590 - avme</modelNumber>
<modelURL>www.avm.de</modelURL>
<UDN>uuid:75802409-bccb-40e7-8e69-04B4FE974B47</UDN>
<UPC>AVM TR-064</UPC>
<serviceList>
<service>
<serviceType>urn:dslforum-org:service:X_AVM-DE_WANMobileConnection:1</serviceType>
<serviceId>urn:X_AVM-DE_WANMobileConnection-com:serviceId:X_AVM-DE_WANMobileConnection1</serviceId>
<controlURL>/upnp/control/x_wanmobileconn</controlURL>
<eventSubURL>/upnp/control/x_wanmobileconn</eventSubURL>
<SCPDURL>/x_wanmobileconnSCPD.xml</SCPDURL>
</service>
<service>
<serviceType>urn:dslforum-org:service:X_AVM-DE_WANFiber:1</serviceType>
<serviceId>urn:X_AVM-DE_WANFiber-com:serviceId:X_AVM-DE_WANFiber1</serviceId>
<controlURL>/upnp/control/x_wanfiber</controlURL>
<eventSubURL>/upnp/control/x_wanfiber</eventSubURL>
<SCPDURL>/x_wanfiberSCPD.xml</SCPDURL>
</service>

C:\Users\micro>

[desosav]

attached you can find the curl output. As expected root is ommited. @JSoko absolutely. I will report this later today :)

curl_output.txt

[desosav]

a new beta version has been released earlier today. I am at work and unable to apply the update at the moment. @JSoko Maybe you can test the new build and report back if it resolves the issue. I will share my feedback as soon as i am able to update my fritz.

[kbr]

I've reported the issue this morning and hope AVM can reproduce and fix it at least for the final OS 7.70 version. I suppose that a test with curl would be sufficient. If curl returns valid xml-data it should also work with requests.

[desosav]

The latest beta release has fixed the issue.

[JSoko]

Issue is fixed with Fritz!Os 7.70-108716 from 10/17/2023.

Thanks for Your help!