search

kdschlosser / UPNP_Device

Discovers UPNP devices on the network.
GNU Lesser General Public License v3.0
3 stars 1 forks source link

Some requests return non-XML data and exception on parsing XML #7

Open raydog153 opened 5 years ago

raydog153 commented 5 years ago

I see in service.py you have try/except around parsing response as xml. Some endpoints do not return xml, applying same fix within upnp_class.py resolves this error.

Exception:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/Users/rboutotte/samsungtv/UPNP_Device/upnp_device/__main__.py", line 87, in do
    for device in UPNP_Device.discover(args.timeout, log_level, args.ips, args.dump):
  File "/usr/local/lib/python2.7/site-packages/UPNP_Device-0.1.0b0-py2.7.egg/UPNP_Device/__init__.py", line 20, in discover
    yield UPNPObject(addr, locations, dump)
  File "/usr/local/lib/python2.7/site-packages/UPNP_Device-0.1.0b0-py2.7.egg/UPNP_Device/instance_singleton.py", line 11, in __call__
    super(InstanceSingleton, cls).__call__(id, *args)
  File "/usr/local/lib/python2.7/site-packages/UPNP_Device-0.1.0b0-py2.7.egg/UPNP_Device/upnp_class.py", line 66, in __init__
    root = etree.fromstring(response.content)
  File "src/lxml/etree.pyx", line 3211, in lxml.etree.fromstring
  File "src/lxml/parser.pxi", line 1877, in lxml.etree._parseMemoryDocument
  File "src/lxml/parser.pxi", line 1765, in lxml.etree._parseDoc
  File "src/lxml/parser.pxi", line 1127, in lxml.etree._BaseParser._parseDoc
  File "src/lxml/parser.pxi", line 601, in lxml.etree._ParserContext._handleParseResultDoc
  File "src/lxml/parser.pxi", line 711, in lxml.etree._handleParseResult
  File "src/lxml/parser.pxi", line 640, in lxml.etree._raiseParseError
XMLSyntaxError: Start tag expected, '<' not found, line 1, column 1 (line 1)

This is the core fix, however I also changed the logic around to parse using content rather than response.content since there was some extra processing done for content to handle I think python 2.7 vs 3 wrt string encoding/decoding differences, so this made sense:

            try:
                root = etree.fromstring(content)
            except:
                import traceback
                print('Content response is not XML, url: ' + url + location)
                print('Dump of non-XML:')
                print(repr(response.content))
                traceback.print_exc()
                continue
kdschlosser commented 5 years ago

I need to see dump files.

The biggest problem with UPNP is even tho there is a standard. it seems that no one follows it. so we have to deal with each device issue and code around the issue to produce the results we want. and we have to do it in a manner that is not going to screw it up for another device.

raydog153 commented 5 years ago

Agreed, hence why I am going thru each of my 8 devices to make sure they work, and when I fix something I try on other devices.

One response returns this and which clearly is not XML:

status=ok

Screenshot showing bytes returned and content type is json:

screen shot 2019-01-25 at 8 52 39 am
raydog153 commented 5 years ago

SSDP details:

{
    "OPT": "\"http://schemas.upnp.org/upnp/1/0/\"; ns=01", 
    "X-ACCEPTS-REGISTRATION": "3", 
    "X-MDX-CAPS": "", 
    "X-FRIENDLY-NAME": "TWFzdGVyIGJlZHJvb20=", 
    "X-MDX-ID": "32b500f2d778f8fca8752c5f2c4fdee4", 
    "USN": "uuid:RKU-381XX-YH001G188787::upnp:rootdevice", 
    "X-USER-AGENT": "NRDP MDX", 
    "SERVER": "Linux/3.10.108-grsec-rt122, UPnP/1.0, Portable SDK for UPnP devices/1.6.13", 
    "EXT": "", 
    "ST": "upnp:rootdevice", 
    "01-NLS": "1b9fd652-2039-11e9-8545-b54372f9174b", 
    "X-MSL": "1", 
    "CACHE-CONTROL": "max-age=1800", 
    "DATE": "Fri, 25 Jan 2019 02:04:42 GMT", 
    "X-MDX-REGISTERED": "1", 
    "TYPE": "response", 
    "LOCATION": "http://192.168.1.143:9080"
}
{
    "USN": "uuid:29600001-5c02-10e1-8073-c83a6bd59919::upnp:rootdevice", 
    "ST": "upnp:rootdevice", 
    "EXT": "", 
    "LOCATION": "http://192.168.1.143:8060/", 
    "CACHE-CONTROL": "max-age=3600", 
    "WAKEUP": "MAC=c8:3a:6b:d5:99:19;Timeout=10", 
    "SERVER": "Roku UPnP/1.0 Roku/9.0.0", 
    "TYPE": "response"
}
raydog153 commented 5 years ago

And showing details of working request is not json content type:

screen shot 2019-01-25 at 8 54 18 am
kdschlosser commented 5 years ago

whatever those 2 devices are that are responding are not returning a UPNP spec'd location.

http://192.168.1.143:8060/
http://192.168.1.143:9080

you see the variation in the 2 replies???

one has a slash the other doesn't that's problem 1. but neither of them give you a file name.

try connecting to them with your browser. and paste the output.. and we are not looking for JSON, we are looking for XML.

raydog153 commented 5 years ago

Yes I get files named '.xml' that have no base filename. If you look the LOCATION urls have no path from the SSDP details.

Understood looking for xml, but that is not what is being returned. The trailing slash should not make a difference and making the request either way I get same results.

Even more odd is that I have 2 Roku devices, one is '+' model other is not. One device does not have this issue because it does not report SSDP location under port 9080.

Bah, and now it no longer finds port 9080. Perhaps it has to do with it's on/off state? Will have to test any difference wrt powered state tonight.

Regardless the output from browser I showed in comment https://github.com/kdschlosser/UPNP_Device/issues/7#issuecomment-457578804 , and can see the request URL, response headers, and content length.

So what would be a better fix? If the LOCATION URL has no path then should the device be ignored? If not then how best to fix? Surely due to lack of standards we should protect against UPNP device not returning XML, agreed?

raydog153 commented 5 years ago

I've confirmed that the both my roku devices have this endpoint when powered on, but if off the endpoint does not exist. And in both cases it is not returning XML.

My other roku device http://192.168.1.35:9080 and output of curl:

rboutotte@MacBook-Pro-2:~/samsungtv/UPNP_Device$ curl http://192.168.1.35:9080
status=ok
rboutotte@MacBook-Pro-2:~/samsungtv/UPNP_Device$ curl -v http://192.168.1.35:9080
* Rebuilt URL to: http://192.168.1.35:9080/
*   Trying 192.168.1.35...
* TCP_NODELAY set
* Connected to 192.168.1.35 (192.168.1.35) port 9080 (#0)
> GET / HTTP/1.1
> Host: 192.168.1.35:9080
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Access-Control-Allow-Origin: *
< Cache: no-cache
< Content-Type: application/json
< Connection: close
< Content-Length: 9
< Date: Fri Jan 25 11:52:46 2019
<
* Closing connection 0
status=ok
kdschlosser commented 5 years ago

if the unit is powered off then it should not respond to SSDP Discover packets either.

kdschlosser commented 5 years ago

I have never seen this issue at the bottom level. because a device should not respond to discover packets if it is powered off. I have filtered this issue elsewhere in the script. Just not in the UPNPObject class.

So what I need to know is this I guess...

are the SSDP packets specifically the LOCATION different when the unit is powered on vs powered off??

raydog153 commented 5 years ago

Let me correct my wording. Not hard powered off, but Roku is connected to TV HDMI, and NOT powered thru TV but thru outlet, so if TV is off Roku is always on.

By powered off I really mean TV is off and Roku is in standby I guess. There is not even a standby mode, but I think it auto detects TV input is off and goes into standby.

Or perhaps it is not standby but when it is streaming that the endpoint exists.

Doesn't really matter, point being is that based on state of device it exposes additional SSDP endpoints.

kdschlosser commented 5 years ago

ok cool it does.

I can urlparse then to break down the LOCATION and if there is no endpoint supplied then not build a UPNPObject from it.

kdschlosser commented 5 years ago

Powered off these days is assumed to be standby. any device that has a remote that you can press a button on to turn it on. never actually powers down fully.

raydog153 commented 5 years ago

True....but devil is in the details...can depend on how things get hooked up. i.e. if I did use TV power for Roku then it would truly be powered down.

Question, why is the same SSDP endpoint repeated over and over here:

Finding UPNP Devices please wait..
SSDP: 239.255.255.250
M-SEARCH * HTTP/1.1
ST: upnp:rootdevice
MAN: "ssdp:discover"
HOST: 239.255.255.250:1900
MX: 1
Content-Length: 0

SSDP: 192.168.1.35
M-SEARCH * HTTP/1.1
ST: upnp:rootdevice
MAN: "ssdp:discover"
HOST: 239.255.255.250:1900
MX: 1
Content-Length: 0

SSDP: inbound packet for IP 192.168.1.35
{
    "OPT": "\"http://schemas.upnp.org/upnp/1/0/\"; ns=01",
    "X-ACCEPTS-REGISTRATION": "3",
    "X-MDX-CAPS": "",
    "X-FRIENDLY-NAME": "TGl2aW5nIHJvb20=",
    "USN": "uuid:RKU-466XX-YJ00DY776259::upnp:rootdevice",
    "X-USER-AGENT": "NRDP MDX",
    "SERVER": "Linux/3.10.108-grsec-rt122, UPnP/1.0, Portable SDK for UPnP devices/1.6.13",
    "EXT": "",
    "ST": "upnp:rootdevice",
    "01-NLS": "b09e7ed0-20bf-11e9-8624-c5d0561b2d06",
    "X-MSL": "1",
    "CACHE-CONTROL": "max-age=1800",
    "DATE": "Fri, 25 Jan 2019 17:00:55 GMT",
    "X-MDX-REGISTERED": "1",
    "TYPE": "response",
    "LOCATION": "http://192.168.1.35:9080"
}
SSDP: inbound packet for IP 192.168.1.35
{
    "OPT": "\"http://schemas.upnp.org/upnp/1/0/\"; ns=01",
    "X-ACCEPTS-REGISTRATION": "3",
    "X-MDX-CAPS": "",
    "X-FRIENDLY-NAME": "TGl2aW5nIHJvb20=",
    "USN": "uuid:RKU-466XX-YJ00DY776259::upnp:rootdevice",
    "X-USER-AGENT": "NRDP MDX",
    "SERVER": "Linux/3.10.108-grsec-rt122, UPnP/1.0, Portable SDK for UPnP devices/1.6.13",
    "EXT": "",
    "ST": "upnp:rootdevice",
    "01-NLS": "b09e7ed0-20bf-11e9-8624-c5d0561b2d06",
    "X-MSL": "1",
    "CACHE-CONTROL": "max-age=1800",
    "DATE": "Fri, 25 Jan 2019 17:00:55 GMT",
    "X-MDX-REGISTERED": "1",
    "TYPE": "response",
    "LOCATION": "http://192.168.1.35:9080"
}
SSDP: inbound packet for IP 192.168.1.35
{
    "OPT": "\"http://schemas.upnp.org/upnp/1/0/\"; ns=01",
    "X-ACCEPTS-REGISTRATION": "3",
    "X-MDX-CAPS": "",
    "X-FRIENDLY-NAME": "TGl2aW5nIHJvb20=",
    "USN": "uuid:RKU-466XX-YJ00DY776259::upnp:rootdevice",
    "X-USER-AGENT": "NRDP MDX",
    "SERVER": "Linux/3.10.108-grsec-rt122, UPnP/1.0, Portable SDK for UPnP devices/1.6.13",
    "EXT": "",
    "ST": "upnp:rootdevice",
    "01-NLS": "b09e7ed0-20bf-11e9-8624-c5d0561b2d06",
    "X-MSL": "1",
    "CACHE-CONTROL": "max-age=1800",
    "DATE": "Fri, 25 Jan 2019 17:00:55 GMT",
    "X-MDX-REGISTERED": "1",
    "TYPE": "response",
    "LOCATION": "http://192.168.1.35:9080"
}
SSDP: inbound packet for IP 192.168.1.35
{
    "OPT": "\"http://schemas.upnp.org/upnp/1/0/\"; ns=01",
    "X-ACCEPTS-REGISTRATION": "3",
    "X-MDX-CAPS": "",
    "X-FRIENDLY-NAME": "TGl2aW5nIHJvb20=",
    "USN": "uuid:RKU-466XX-YJ00DY776259::upnp:rootdevice",
    "X-USER-AGENT": "NRDP MDX",
    "SERVER": "Linux/3.10.108-grsec-rt122, UPnP/1.0, Portable SDK for UPnP devices/1.6.13",
    "EXT": "",
    "ST": "upnp:rootdevice",
    "01-NLS": "b09e7ed0-20bf-11e9-8624-c5d0561b2d06",
    "X-MSL": "1",
    "CACHE-CONTROL": "max-age=1800",
    "DATE": "Fri, 25 Jan 2019 17:00:55 GMT",
    "X-MDX-REGISTERED": "1",
    "TYPE": "response",
    "LOCATION": "http://192.168.1.35:9080"
}
SSDP: inbound packet for IP 192.168.1.35
{
    "OPT": "\"http://schemas.upnp.org/upnp/1/0/\"; ns=01",
    "X-ACCEPTS-REGISTRATION": "3",
    "X-MDX-CAPS": "",
    "X-FRIENDLY-NAME": "TGl2aW5nIHJvb20=",
    "USN": "uuid:RKU-466XX-YJ00DY776259::upnp:rootdevice",
    "X-USER-AGENT": "NRDP MDX",
    "SERVER": "Linux/3.10.108-grsec-rt122, UPnP/1.0, Portable SDK for UPnP devices/1.6.13",
    "EXT": "",
    "ST": "upnp:rootdevice",
    "01-NLS": "b09e7ed0-20bf-11e9-8624-c5d0561b2d06",
    "X-MSL": "1",
    "CACHE-CONTROL": "max-age=1800",
    "DATE": "Fri, 25 Jan 2019 17:00:55 GMT",
    "X-MDX-REGISTERED": "1",
    "TYPE": "response",
    "LOCATION": "http://192.168.1.35:9080"
}
SSDP: inbound packet for IP 192.168.1.35
{
    "OPT": "\"http://schemas.upnp.org/upnp/1/0/\"; ns=01",
    "X-ACCEPTS-REGISTRATION": "3",
    "X-MDX-CAPS": "",
    "X-FRIENDLY-NAME": "TGl2aW5nIHJvb20=",
    "USN": "uuid:RKU-466XX-YJ00DY776259::upnp:rootdevice",
    "X-USER-AGENT": "NRDP MDX",
    "SERVER": "Linux/3.10.108-grsec-rt122, UPnP/1.0, Portable SDK for UPnP devices/1.6.13",
    "EXT": "",
    "ST": "upnp:rootdevice",
    "01-NLS": "b09e7ed0-20bf-11e9-8624-c5d0561b2d06",
    "X-MSL": "1",
    "CACHE-CONTROL": "max-age=1800",
    "DATE": "Fri, 25 Jan 2019 17:00:55 GMT",
    "X-MDX-REGISTERED": "1",
    "TYPE": "response",
    "LOCATION": "http://192.168.1.35:9080"
}
SSDP: inbound packet for IP 192.168.1.35
{
    "OPT": "\"http://schemas.upnp.org/upnp/1/0/\"; ns=01",
    "X-ACCEPTS-REGISTRATION": "3",
    "X-MDX-CAPS": "",
    "X-FRIENDLY-NAME": "TGl2aW5nIHJvb20=",
    "USN": "uuid:RKU-466XX-YJ00DY776259::upnp:rootdevice",
    "X-USER-AGENT": "NRDP MDX",
    "SERVER": "Linux/3.10.108-grsec-rt122, UPnP/1.0, Portable SDK for UPnP devices/1.6.13",
    "EXT": "",
    "ST": "upnp:rootdevice",
    "01-NLS": "b09e7ed0-20bf-11e9-8624-c5d0561b2d06",
    "X-MSL": "1",
    "CACHE-CONTROL": "max-age=1800",
    "DATE": "Fri, 25 Jan 2019 17:00:55 GMT",
    "X-MDX-REGISTERED": "1",
    "TYPE": "response",
    "LOCATION": "http://192.168.1.35:9080"
}
SSDP: inbound packet for IP 192.168.1.35
{
    "OPT": "\"http://schemas.upnp.org/upnp/1/0/\"; ns=01",
    "X-ACCEPTS-REGISTRATION": "3",
    "X-MDX-CAPS": "",
    "X-FRIENDLY-NAME": "TGl2aW5nIHJvb20=",
    "USN": "uuid:RKU-466XX-YJ00DY776259::upnp:rootdevice",
    "X-USER-AGENT": "NRDP MDX",
    "SERVER": "Linux/3.10.108-grsec-rt122, UPnP/1.0, Portable SDK for UPnP devices/1.6.13",
    "EXT": "",
    "ST": "upnp:rootdevice",
    "01-NLS": "b09e7ed0-20bf-11e9-8624-c5d0561b2d06",
    "X-MSL": "1",
    "CACHE-CONTROL": "max-age=1800",
    "DATE": "Fri, 25 Jan 2019 17:00:55 GMT",
    "X-MDX-REGISTERED": "1",
    "TYPE": "response",
    "LOCATION": "http://192.168.1.35:9080"
}
SSDP: inbound packet for IP 192.168.1.35
{
    "OPT": "\"http://schemas.upnp.org/upnp/1/0/\"; ns=01",
    "X-ACCEPTS-REGISTRATION": "3",
    "X-MDX-CAPS": "",
    "X-FRIENDLY-NAME": "TGl2aW5nIHJvb20=",
    "USN": "uuid:RKU-466XX-YJ00DY776259::upnp:rootdevice",
    "X-USER-AGENT": "NRDP MDX",
    "SERVER": "Linux/3.10.108-grsec-rt122, UPnP/1.0, Portable SDK for UPnP devices/1.6.13",
    "EXT": "",
    "ST": "upnp:rootdevice",
    "01-NLS": "b09e7ed0-20bf-11e9-8624-c5d0561b2d06",
    "X-MSL": "1",
    "CACHE-CONTROL": "max-age=1800",
    "DATE": "Fri, 25 Jan 2019 17:00:55 GMT",
    "X-MDX-REGISTERED": "1",
    "TYPE": "response",
    "LOCATION": "http://192.168.1.35:9080"
}
SSDP: inbound packet for IP 192.168.1.35
{
    "OPT": "\"http://schemas.upnp.org/upnp/1/0/\"; ns=01",
    "X-ACCEPTS-REGISTRATION": "3",
    "X-MDX-CAPS": "",
    "X-FRIENDLY-NAME": "TGl2aW5nIHJvb20=",
    "USN": "uuid:RKU-466XX-YJ00DY776259::upnp:rootdevice",
    "X-USER-AGENT": "NRDP MDX",
    "SERVER": "Linux/3.10.108-grsec-rt122, UPnP/1.0, Portable SDK for UPnP devices/1.6.13",
    "EXT": "",
    "ST": "upnp:rootdevice",
    "01-NLS": "b09e7ed0-20bf-11e9-8624-c5d0561b2d06",
    "X-MSL": "1",
    "CACHE-CONTROL": "max-age=1800",
    "DATE": "Fri, 25 Jan 2019 17:00:55 GMT",
    "X-MDX-REGISTERED": "1",
    "TYPE": "response",
    "LOCATION": "http://192.168.1.35:9080"
}
SSDP: inbound packet for IP 192.168.1.35
{
    "USN": "uuid:29680014-a40b-10d8-8043-c83a6b1740d8::upnp:rootdevice",
    "ST": "upnp:rootdevice",
    "EXT": "",
    "LOCATION": "http://192.168.1.35:8060/",
    "CACHE-CONTROL": "max-age=3600",
    "WAKEUP": "MAC=c8:3a:6b:17:40:d9;Timeout=10",
    "SERVER": "Roku UPnP/1.0 Roku/9.0.0",
    "TYPE": "response"
}
raydog153 commented 5 years ago

If you notice the "http://192.168.1.35:9080" endpoint is 10 times, other endpoint once.

kdschlosser commented 5 years ago

is this a valid endpoint?

http://192.168.1.35:8060/
raydog153 commented 5 years ago

Yes it is. Doesn't return much thou.

Curl output:

rboutotte@MacBook-Pro-2:~/Downloads$ curl -v http://192.168.1.35:8060/
*   Trying 192.168.1.35...
* TCP_NODELAY set
* Connected to 192.168.1.35 (192.168.1.35) port 8060 (#0)
> GET / HTTP/1.1
> Host: 192.168.1.35:8060
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: Roku UPnP/1.0 MiniUPnPd/1.4
< Content-Length: 1142
< Cache-Control: no-cache
< Content-Type: text/xml; charset="utf-8"
<
<?xml version="1.0"?>
<root xmlns="urn:schemas-upnp-org:device-1-0">
<specVersion>
<major>1</major>
<minor>0</minor>
</specVersion>
<device>
<deviceType>urn:roku-com:device:player:1-0</deviceType>
<friendlyName>Living room</friendlyName>
<manufacturer>Roku</manufacturer>
<manufacturerURL>http://www.roku.com/</manufacturerURL>
<modelDescription>Roku Streaming Player Network Media</modelDescription>
<modelName>Roku Ultra</modelName>
<modelNumber>4660X</modelNumber>
<modelURL>http://www.roku.com/</modelURL>
<serialNumber>YJ00DY776259</serialNumber>
<UDN>uuid:29680014-a40b-10d8-8043-c83a6b1740d8</UDN>
<serviceList>
<service>
<serviceType>urn:roku-com:service:ecp:1</serviceType>
<serviceId>urn:roku-com:serviceId:ecp1-0</serviceId>
<controlURL></controlURL>
<eventSubURL></eventSubURL>
<SCPDURL>ecp_SCPD.xml</SCPDURL>
</service>
<service>
<serviceType>urn:dial-multiscreen-org:service:dial:1</serviceType>
<serviceId>urn:dial-multiscreen-org:serviceId:dial1-0</serviceId>
<controlURL></controlURL>
<eventSubURL></eventSubURL>
<SCPDURL>dial_SCPD.xml</SCPDURL>
</service>
</serviceList>
</device>
</root>
* Connection #0 to host 192.168.1.35 left intact
rboutotte@MacBook-Pro-2:~/Downloads$