search

xbmc / xbmc

Kodi is an award-winning free and open source home theater/media center software and entertainment hub for digital media. With its beautiful interface and powerful skinning engine, it's available for Android, BSD, Linux, macOS, iOS, tvOS and Windows.
https://kodi.tv/
Other
18.42k stars 6.3k forks source link

xbmcvfs.File().read() no longer works with http/s urls since 3/21 #16013

Open primaeval opened 5 years ago

primaeval commented 5 years ago

Bug report

Describe the bug

Here is a clear and concise description of what the problem is:

xbmcvfs.File().read() in python addons has ceased to work with http/s urls since 3/21

Here is an example that I have had to patch to use requests.get instead. https://github.com/primaeval/plugin.program.xmltv.meld/commit/9c5e79e907d87bb19a1f8bdb9e868056af8105d7

Here is a user report testing various builds. It fails on 3/21. https://forum.kodi.tv/showthread.php?tid=330668&pid=2850460#pid2850460

Expected Behavior

Here is a clear and concise description of what was expected to happen:

xbmcvfs.File().read() can read http urls

Actual Behavior

nothing is returned from read()

Possible Fix

Revert changes to xmbcvfs.File().read()

To Reproduce

Steps to reproduce the behavior:

  1. data = xbmcvfs.File("https://tvlistings.gracenote.com/gapzap_webapi/api/Providers/getPostalCodeProviders/USA/90210/gapzap/en").read()
  2. print repr(data)

Debuglog

The debuglog can be found here: https://paste.kodi.tv/vefiqonazo.kodi

Screenshots

Here are some links or screenshots to help explain the problem:

Additional context or screenshots (if appropriate)

Here is some additional context or explanation that might help:

Your Environment

Used Operating system:

note: Once the issue is made we require you to update it with new information or Kodi versions should that be required. Team Kodi will consider your problem report however, we will not make any promises the problem will be solved.

xbmc-gh-bot[bot] commented 5 years ago

Thank you for using Kodi and our issue tracker. This is your friendly Kodi GitHub bot :wink:

The issue that you have opened was marked as ignoring the rules of our bug tracker by a team member. Please review the following requirements for issue reports and make sure that you follow all of them:

Please understand that adherence to these rules is required for the team to be able to handle the volume of open issues efficiently. The issue will be closed after one week has passed without satisfactory follow-up from your side.

primaeval commented 5 years ago

@pkerling give me a clue what is missing please.

yol commented 5 years ago

Your log is not a debug log, so it does not provide the information we need. Please read the provided link. While you're at it, please enable curl component logging and don't copy-paste the whole log into GitHub, it makes the issue unreadable :-)

In the meantime, thanks for the analysis so far. Could you reproduce on other platforms?

primaeval commented 5 years ago

There you go. Curl debug logging is enabled. I reverted to line 1706 to produce the log. sources = xbmcvfs.File(url,"r").read() https://github.com/primaeval/plugin.program.xmltv.meld/commit/9c5e79e907d87bb19a1f8bdb9e868056af8105d7#diff-5bc02cefb3ea9e27f1a6776eabd1935dL1706

I believe someone else tested it on Android but I can't confirm that myself.

yol commented 5 years ago

15644 seems the most likely culprit so far. Can you try requesting http://........|acceptencoding=?

primaeval commented 5 years ago

acceptencoding=* seems to work
line 1708

    url = 'https://tvlistings.gracenote.com/gapzap_webapi/api/Providers/getPostalCodeProviders/' + country + '/' + zipcode + '/gapzap/en|acceptencoding=*'
    #log(url)
    sources = xbmcvfs.File(url,"r").read()

That is going to be a pain to fix. I've probably made a few dozen addons that rely on it.

yol commented 5 years ago

Don't change your code - we'll try to see what's wrong on the Kodi side. Your code should work as-is. This was only a test to check if this is indeed related.

@candrews Can you investigate?

yol commented 5 years ago

@primaeval Did you also try acceptencoding=? I.e., without the star? It should not be necessary.

primaeval commented 5 years ago

Yes. This is from acceptencoding=? which fails https://paste.kodi.tv/inapudubof.kodi and this is where acceptencoding=* which works https://paste.kodi.tv/tazoletine.kodi

yol commented 5 years ago

Sorry, I realize I have been not completely clear. I meant without anything after the equal sign - this will set it to an empty value:

|acceptencoding=

primaeval commented 5 years ago

I see. |acceptencoding= works. https://paste.kodi.tv/fudewumane.kodi

yol commented 5 years ago

Thanks! Then this pretty much confirms that xbmcvfs is unable to deal with the encoding for some reason.

candrews commented 5 years ago

I noticed that http://tvlistings.gracenote.com redirects to https://tvlistings.gracenote.com - I suspect this may be a component of the problem. Can you please try again with the changes I suggest in https://github.com/primaeval/plugin.program.xmltv.meld/pull/3 ?

primaeval commented 5 years ago

Thanks for the help but it is definitely the acceptencoding that solves it.

If you are testing it the sequence to generate the problem is Kodi \ Video \ Addons \ xmltv Meld \ zap \ USA 1 \ <xmltv Meld error - Check the log>

This is the code on line 1706 with just http, no acceptencoding and it fails with the following log.

    url = 'http://tvlistings.gracenote.com/gapzap_webapi/api/Providers/getPostalCodeProviders/' + country + '/' + zipcode + '/gapzap/en'
    #log(url)
    sources = xbmcvfs.File(url,"r").read()

https://paste.kodi.tv/quwihebuqi.kodi

This is the code on line 1706 with http, acceptencoding= and it passes with the following log.

    url = 'http://tvlistings.gracenote.com/gapzap_webapi/api/Providers/getPostalCodeProviders/' + country + '/' + zipcode + '/gapzap/en|acceptencoding='
    #log(url)
    sources = xbmcvfs.File(url,"r").read()

https://paste.kodi.tv/uqegecezaj.kodi

candrews commented 5 years ago

I'm not arguing that acceptencoding works around the problem - but I am wondering if changing to https also works around it.

I'm wondering if there's a problem with encoding+redirects.

primaeval commented 5 years ago

There was a problem with one of the gracenote urls using http or https at one point in the past. It was probably the one I changed to https. Maybe they had a server down. It seems to be ok either way today.

candrews commented 5 years ago

So does using the https url without acceptencoding work?

primaeval commented 5 years ago

No.

    url = 'https://tvlistings.gracenote.com/gapzap_webapi/api/Providers/getPostalCodeProviders/' + country + '/' + zipcode + '/gapzap/en'
    #log(url)
    sources = xbmcvfs.File(url,"r").read()

https://paste.kodi.tv/kovibosonu.kodi

candrews commented 5 years ago

Here's some more info. This command produces the same request and response headers as your kodi logs, but it works (the json is returned, unlike the error you experience in kodi)

$ curl https://tvlistings.gracenote.com/gapzap_webapi/api/Providers/getPostalCodeProviders/USA/90210/gapzap/en -H "Accept-Charset: UTF-8,*;q=0.8" --compressed --range 0- --http1.1 -v
*   Trying 35.160.15.77...
* TCP_NODELAY set
* Connected to tvlistings.gracenote.com (35.160.15.77) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=tvlistings.gracenote.com
*  start date: Jul 24 00:00:00 2018 GMT
*  expire date: Aug 24 12:00:00 2019 GMT
*  subjectAltName: host "tvlistings.gracenote.com" matched cert's "tvlistings.gracenote.com"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
> GET /gapzap_webapi/api/Providers/getPostalCodeProviders/USA/90210/gapzap/en HTTP/1.1
> Host: tvlistings.gracenote.com
> Range: bytes=0-
> User-Agent: curl/7.64.0
> Accept: */*
> Accept-Encoding: deflate, gzip, br
> Accept-Charset: UTF-8,*;q=0.8
> 
< HTTP/1.1 206 Partial Content
< Date: Tue, 30 Apr 2019 19:29:50 GMT
< Content-Type: application/json; charset=utf-8
< Content-Length: 647
< Connection: keep-alive
< Server: Apache
< Cache-Control: no-cache
< Expires: -1
< Pragma: no-cache
< Vary: Accept-Encoding
< Content-Encoding: gzip
< Access-Control-Allow-Origin: *
< Content-Range: bytes 0-646/647
<
primaeval commented 5 years ago

Was this the only curl related change to kodi just before 3/21 ? https://github.com/xbmc/xbmc/pull/15644

If it is then it probably is the very act of setting CURLOPT_ACCEPT_ENCODING, even if it is the empty string.

The only other change is the removal of the SetAcceptEncoding gzip.

yol commented 5 years ago

If it is then it probably is the very act of setting CURLOPT_ACCEPT_ENCODING, even if it is the empty string.

Most probably. Still there is no reason the Python interface shouldn't be able to handle that, so it's a bug. I'm willing to bet that using e.g. http://....|acceptencoding=gzip wouldn't have worked before as well.

github-actions[bot] commented 2 months ago

This issue is now marked stale because it has been open over a year without activity. Remove the stale label or add a comment to reset the stale state.