TV Guide doesn't work anymore

vermope commented 4 years ago

Describe the bug Service IPTV manager gives an error when fetching program schedule for all VTM GO channels. For the VTM GO channels there is no program data in the EPG.

To Reproduce Steps to reproduce the behavior:

Go to VTM GO settings --> Integrations --> IPTV manager settings
Channels --> Refresh channels and guide now...
VTM GO won't load new program data

Expected behavior New program schedule should fill the EPG automatically on a daily base

Screenshots If applicable, add screenshots to help explain your problem.

System

Addon version: VTM GO -> 1.1.2 / service.iptv.manager -> 0.1.0
Kodi version: 18.8
Inputstream adaptive version:
Operating System: LibreElec 9.1.4
Special Hardware: X86/X64

michaelarnauts commented 4 years ago

Service IPTV manager gives an error when fetching program schedule for all VTM GO channels. For the VTM GO channels there is no program data in the EPG.

Can you indicate what error you get and provide the relevant parts of your kodi.log? As far as I can tell, this should work fine.

vermope commented 4 years ago

See log here: https://pastebin.ubuntu.com/p/dG55QtvcTN/

michaelarnauts commented 4 years ago

Krijg je ook een error als je vanuit de addon naar een kanaal gaat en daar de gids probeert te bekijken?

vermope commented 4 years ago

Jup,

2020-10-01 16:05:07.931 T:139976714426112 ERROR: GetDirectory - Error getting plugin://plugin.video.vtm.go/tvguide/channel/vtm/today 2020-10-01 16:05:07.942 T:139979768330368 ERROR: CGUIMediaWindow::GetDirectory(plugin://plugin.video.vtm.go/tvguide/channel/vtm/today) failed 2020-10-01 16:05:13.343 T:139976669308672 ERROR: EXCEPTION Thrown (PythonToCppException) : -->Python callback/script returned the following error<--

NOTE: IGNORING THIS CAN LEAD TO MEMORY LEAKS! Error Type: <type 'exceptions.ValueError'> Error Contents: No JSON object could be decoded Traceback (most recent call last): File "/storage/.kodi/addons/plugin.video.vtm.go/plugin.py", line 18, in plugin.run(sys.argv) File "/storage/.kodi/addons/plugin.video.vtm.go/resources/lib/plugin.py", line 218, in run routing.run(params) File "/storage/.kodi/addons/script.module.routing/lib/routing.py", line 130, in run self._dispatch(self.path) File "/storage/.kodi/addons/script.module.routing/lib/routing.py", line 141, in _dispatch view_func(**kwargs) File "/storage/.kodi/addons/plugin.video.vtm.go/resources/lib/plugin.py", line 72, in show_tvguide_detail TvGuide(kodi).show_tvguide_detail(channel, date) File "/storage/.kodi/addons/plugin.video.vtm.go/resources/lib/modules/tvguide.py", line 57, in show_tvguide_detail epg = self._vtm_go_epg.get_epg(channel=channel, date=date) File "/storage/.kodi/addons/plugin.video.vtm.go/resources/lib/vtmgo/vtmgoepg.py", line 106, in get_epg epg = json.loads(response) File "/usr/lib/python2.7/json/init.py", line 339, in loads File "/usr/lib/python2.7/json/decoder.py", line 364, in decode File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode ValueError: No JSON object could be decoded -->End of Python script error report<-- 2020-10-01 16:05:13.745 T:139976358012672 ERROR: GetDirectory - Error getting plugin://plugin.video.vtm.go/tvguide/channel/vtm/tomorrow 2020-10-01 16:05:13.760 T:139979768330368 ERROR: CGUIMediaWindow::GetDirectory(plugin://plugin.video.vtm.go/tvguide/channel/vtm/tomorrow) failed

michaelarnauts commented 4 years ago

Hmm, ik heb het hier ook voor. Ik vermoed dat de API aangepast is bij VTM.

canihavesomecoffee commented 4 years ago

Dat zou het meest logische zijn. Hetzelfde probleem speelt hier al een aantal dagen. Stacktrace is iets minder uitgebreid, maar komt op hetzelfde neer:

2020-10-01 19:19:07.936 T:7442   ERROR: [service.iptv.manager] [resources.lib.modules.addon] Something went wrong while calling plugin.video.vtm.go: Something went wrong in plugin.video.vtm.go
2020-10-01 19:19:07.968 T:7775   ERROR: EXCEPTION Thrown (PythonToCppException) : -->Python callback/script returned the following error<--
                                             - NOTE: IGNORING THIS CAN LEAD TO MEMORY LEAKS!
                                            Error Type: <type 'exceptions.ValueError'>
                                            Error Contents: No JSON object could be decoded
                                            Traceback (most recent call last):
                                              File "/storage/emulated/0/Android/data/org.xbmc.kodi/files/.kodi/addons/plugin.video.vtm.go/plugin.py", line 18, in <module>
                                                plugin.run(sys.argv)
                                              File "/storage/emulated/0/Android/data/org.xbmc.kodi/files/.kodi/addons/plugin.video.vtm.go/resources/lib/plugin.py", line 218, in run
                                                routing.run(params)
                                              File "/storage/emulated/0/Android/data/org.xbmc.kodi/files/.kodi/addons/script.module.routing/lib/routing.py", line 130, in run
                                                self._dispatch(self.path)
                                              File "/storage/emulated/0/Android/data/org.xbmc.kodi/files/.kodi/addons/script.module.routing/lib/routing.py", line 141, in _dispatch
                                                view_func(**kwargs)
                                              File "/storage/emulated/0/Android/data/org.xbmc.kodi/files/.kodi/addons/plugin.video.vtm.go/resources/lib/plugin.py", line 213, in iptv_epg
                                                IPTVManager(kodi, int(routing.args['port'][0])).send_epg()
                                              File "/storage/emulated/0/Android/data/org.xbmc.kodi/files/.kodi/addons/plugin.video.vtm.go/resources/lib/modules/iptvmanager.py", line 38, in send
                                                sock.sendall(json.dumps(func(self)).encode())  # pylint: disable=not-callable
                                              File "/storage/emulated/0/Android/data/org.xbmc.kodi/files/.kodi/addons/plugin.video.vtm.go/resources/lib/modules/iptvmanager.py", line 78, in send_epg
                                                channels = self._vtm_go_epg.get_epgs(date)
                                              File "/storage/emulated/0/Android/data/org.xbmc.kodi/files/.kodi/addons/plugin.video.vtm.go/resources/lib/vtmgo/vtmgoepg.py", line 133, in get_epgs
                                                epg = json.loads(response)
                                              File "/home/jenkins/workspace/Android-ARM-Leia/tools/depends/xbmc-depends/arm-linux-androideabi-21-release/lib/python2.7/json/__init__.py", line 339, in loads
                                              File "/home/jenkins/workspace/Android-ARM-Leia/tools/depends/xbmc-depends/arm-linux-androideabi-21-release/lib/python2.7/json/decoder.py", line 364, in decode
                                              File "/home/jenkins/workspace/Android-ARM-Leia/tools/depends/xbmc-depends/arm-linux-androideabi-21-release/lib/python2.7/json/decoder.py", line 382, in raw_decode
                                            ValueError: No JSON object could be decoded
                                            -->End of Python script error report<--
2020-10-01 19:19:16.401 T:7792   ERROR: EXCEPTION: Invalid setting type
2020-10-01 19:24:17.852 T:7398   ERROR: Previous line repeats 8 times.

dagwieers commented 4 years ago

Ja, ik heb dat al een paar dagen. Ook al gemeld op Slack, maar ik dacht dat het misschien aan mijn master branch versie lag ofzo.

canihavesomecoffee commented 4 years ago

De URL (i.e. https://vtm.be/tv-gids/api/v2/broadcasts/2020-10-01) lijkt in elk geval vanuit de browser nog steeds te werken.

Via Python niet daarentegen:

import requests
s = requests.session()
s.cookies.set('pws', 'functional|analytics|content_recommendation|targeted_advertising|social_media')
Out[23]: Cookie(version=0, name='pws', value='functional|analytics|content_recommendation|targeted_advertising|social_media', port=None, port_specified=False, domain='', domain_specified=False, domain_initial_dot=False, path='/', path_specified=True, secure=False, expires=None, discard=True, comment=None, comment_url=None, rest={'HttpOnly': None}, rfc2109=False)
s.cookies.set('pwv', '1')
Out[24]: Cookie(version=0, name='pwv', value='1', port=None, port_specified=False, domain='', domain_specified=False, domain_initial_dot=False, path='/', path_specified=True, secure=False, expires=None, discard=True, comment=None, comment_url=None, rest={'HttpOnly': None}, rfc2109=False)
r = s.get('https://vtm.be/tv-gids/api/v2/broadcasts/2020-10-01')
r.text
Out[26]: '<!DOCTYPE html>\n<html lang=\'nl\'>\n<head>\n    <meta name="viewport" content="width=device-width, initial-scale=1.0">\n\n    <style>\n        @font-face {\n            font-family: Sharp Sans Display;\n            font-style: normal;\n            font-weight: 600;\n            /*noinspection CssUnknownTarget*/\n            src: url("/consent/resources/fonts/SharpSansDispNo2-Bold.otf") format("opentype");\n        }\n\n        @font-face {\n            font-family: Montserrat;\n            font-style: normal;\n            font-weight: 400;\n            /*noinspection CssUnknownTarget*/\n            src: url("/consent/resources/fonts/Montserrat-Regular.ttf") format("opentype");\n        }\n\n        h2 {\n            font-family: Sharp Sans Display, Arial, sans-serif;\n            font-size: 26px;\n            text-transform: uppercase;\n        }\n\n        body {\n            margin: 0;\n            font-family: Montserrat, Arial, sans-serif;\n            font-size: 16px;\n        }\n\n        .container {\n            /*noinspection CssUnknownTarget*/\n            background-image: url("/consent/resources/backgrounds/vtm.png");\n            width: 100vw;\n            height: 100vh;\n            background-position: top center;\n            background-size: cover;\n            background-repeat: no-repeat;\n            background-color: #f5f5f5;\n        }\n\n        @media only screen and (max-width: 768px) {\n            .container {\n                /*noinspection CssUnknownTarget*/\n                background-image: url("/consent/resources/backgrounds/medium/vtm.png");\n            }\n        }\n\n        @media only screen and (max-width: 600px) {\n            .container {\n                /*noinspection CssUnknownTarget*/\n                background-image: url("/consent/resources/backgrounds/small/vtm.png");\n            }\n        }\n\n        .modal {\n            position: fixed;\n            background-color: rgb(255, 255, 255);\n            display: flex;\n            flex-direction: column;\n            justify-content: space-between;\n            align-items: center;\n            text-align: center;\n            top: 0;\n            right: 0;\n            bottom: 0;\n            left: 0;\n            height: 74vh;\n            z-index: 999;\n            pointer-events: auto;\n            transition: all 0.5s;\n            visibility: hidden;\n            box-shadow: 0 5px 15px rgba(25, 25, 25, 0.5);\n            max-width: 640px;\n            margin: 13vh auto;\n        }\n\n        .modal__header {\n            flex: 0 10%;\n            width: 100%;\n            display: flex;\n            flex-flow: row nowrap;\n            justify-content: center;\n            align-items: center;\n            border-bottom: #b4b4b4 solid 1px;\n            min-height: 40px;\n            horiz-align: center;\n        }\n\n        .modal__header__logo > img {\n            height: 35px;\n        }\n\n        .modal__header__text {\n            margin-top: 9px;\n            height: 35px;\n            display: flex;\n            justify-content: center;\n            align-items: center;\n        }\n\n        .modal__body {\n            flex: 0 85%;\n            display: flex;\n            flex-flow: column wrap;\n            justify-content: space-evenly;\n            align-items: center;\n        }\n\n        .modal__body__logo {\n            flex: 0 30%;\n            display: flex;\n            flex-flow: column nowrap;\n            justify-content: center;\n            align-items: center;\n        }\n\n        .modal__body__logo > img {\n            height: 86px;\n            max-width: 80%;\n        }\n\n        .modal__body__text {\n            flex: 0 60%\n        }\n\n        .spinner {\n            margin: 64px auto;\n            width: 72px;\n        }\n\n        .spinner__circle {\n            position: relative;\n            width: 72px;\n            height: 72px;\n            overflow: hidden;\n            background-color: #FFB12F;\n            border-radius: 100%;\n\n            -webkit-mask-image: url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAIAAACQd1PeAAAAGXRFWHRTb2Z0d2FyZQBBZG9iZSBJbWFnZVJlYWR5ccllPAAAAA5JREFUeNpiYGBgAAgwAAAEAAGbA+oJAAAAAElFTkSuQmCC);\n            animation: spin 1.5s infinite linear;\n        }\n\n        .spinner__circle-inner, .spinner__circle-gradient {\n            position: absolute;\n            width: 60px;\n            height: 60px;\n            transform: translate(-50%, -50%);\n        }\n\n\n        .spinner__circle-inner {\n            top: 50%;\n            left: 50%;\n            background-color: #fff;\n            border-radius: 100%;\n        }\n\n        .spinner__circle-gradient {\n            top: -24px;\n            left: 6px;\n            background-color: #FFB12F;\n            background-image: -moz-linear-gradient(0deg, #FFB12F 0%, #fff 50%);\n            background-image: -webkit-linear-gradient(0deg, #FFB12F 0%, #fff 50%);\n            background-image: linear-gradient(0deg, #FFB12F 0%, #fff 50%);\n            transform: rotate(-45deg);\n        }\n\n        @keyframes spin {\n            100% {\n                transform: rotate(360deg);\n            }\n        }\n\n        @media (max-width: 960px) {\n            .modal {\n                height: 100vh;\n                width: 100vw;\n                margin: 0;\n                max-width: none;\n            }\n\n            .modal__body {\n                padding: 0 10vw;\n            }\n        }\n    </style>\n    <title>DPG Media Privacy Gate</title>\n\n    <!-- Google Tag Manager -->\n    <script>(function (w, d, s, l, i) {\n        w[l] = w[l] || []\n        w[l].push({\n            \'gtm.start\':\n                new Date().getTime(), event: \'gtm.js\'\n        });\n        var f = d.getElementsByTagName(s)[0],\n            j = d.createElement(s), dl = l != \'dataLayer\' ? \'&l=\' + l : \'\';\n        j.async = true;\n        j.src =\n            \'https://www.googletagmanager.com/gtm.js?id=\' + i + dl;\n        f.parentNode.insertBefore(j, f);\n    })(window, document, \'script\', \'dataLayer\', \'GTM-NT4WR7C\');</script>\n    <!-- End Google Tag Manager -->\n\n    <script type="text/javascript">\n        window.cmpProperties = {\n            privacyManagerId: \'148815\',\n            baseUrl: \'https://vtm.be\',\n            cmpCname: \'https://cmp.dpgmedia.be\',\n            language: \'nl\'\n        }\n    </script>\n    <script type="text/javascript">\n        function getCookie(name) {\n            const value = \'; \' + document.cookie;\n            const parts = value.split(\'; \' + name + \'=\');\n            if (2 === parts.length) return parts.pop().split(\';\').shift();\n        }\n\n        function redirect() {\n            document.getElementById(\'message\').style.visibility = \'visible\';\n            window.location.href = \'https://vtm.be/privacygate-confirm?redirectUri=%2Ftv-gids%2Fapi%2Fv2%2Fbroadcasts%2F2020-10-01&authId=b058ad7e-bc84-4013-b20c-18e50bd4ea34\';\n        }\n\n        function handleError() {\n            document.getElementsByClassName(\'container\')[0].setAttribute(\'data-sourcepoint-error\', \'true\');\n            redirect();\n        }\n\n        const authId = getCookie(\'authId\');\n        window._privacy = [];\n        window._privacy.push([\'functional\', redirect]);\n        window._privacy.push([\'error\', handleError]);\n    </script>\n    <script type="text/javascript" src="https://myprivacy.dpgmedia.be/consent.js"></script>\n</head>\n<body>\n\n<!-- Google Tag Manager (noscript) -->\n<noscript>\n    <iframe title="gtm" src="https://www.googletagmanager.com/ns.html?id=GTM-NT4WR7C"\n            height="0" width="0" style="display:none;visibility:hidden"></iframe>\n</noscript>\n<!-- End Google Tag Manager (noscript) -->\n\n<div class="container">\n    <div id="message" class="modal">\n        <div class="modal__header">\n            <div class="modal__header__logo">\n                <img src="/consent/resources/logos/dpgmedia-logo.svg" alt="dpg media logo"/>\n            </div>\n            <div class="modal__header__text">\n                <span>Privacy</span>\n            </div>\n        </div>\n        <div class="modal__body">\n            <div class="modal__body__logo"><img src="/consent/resources/logos/logo-vtm.svg" alt="VTM"></div>\n            <div class="modal__body__text">\n                <div class="spinner">\n                    <div class="spinner__circle">\n                        <div class="spinner__circle-gradient"></div>\n                        <div class="spinner__circle-inner"></div>\n                    </div>\n                </div>\n            </div>\n        </div>\n    </div>\n</div>\n</body>\n</html>\n'

Lijkt me gerelateerd aan ontbrekende cookies?

De huidige cookies die gezet worden (pws, pwv) lijken geen effect meer te hebben, maar als ik mijn authId cookie meegeef in de sessie krijg ik wel resultaat terug.

michaelarnauts commented 4 years ago

Hmm, het leek te maken te hebben met de cookies. Met een clean session wordt je geredirect en moet je eerst accepteren. Dan krijg je een authId die je meegeeft met de cookies.

Vind het wel vreemd dat je useragent aanpassen ook werkt...

canihavesomecoffee commented 4 years ago

Hmm, het leek te maken te hebben met de cookies. Met een clean session wordt je geredirect en moet je eerst accepteren. Dan krijg je een authId die je meegeeft met de cookies.

Vind het wel vreemd dat je useragent aanpassen ook werkt...

Ik ga nog eens proberen met een totaal andere user agent dan mijn gebruikelijke browser (waar een sessie actief is), ~maar het lijkt er wel op dat het een soort detectie van bots/scrapers is...~

Edit: zal een foutje met de Session langs mijn kant zijn. Console herstarten en het werkt niet meer. Dus is inderdaad de bijgewerkte cookies.

canihavesomecoffee commented 4 years ago

import requests
s = requests.Session()
r = s.get('https://vtm.be/tv-gids/api/v2/broadcasts/2020-10-01')
r.text[:10]
Out[5]: '<!DOCTYPE '
r = s.get('https://vtm.be/privacygate-confirm?redirectUri=%2Ftv-gids%2Fapi%2Fv2%2Fbroadcasts%2F2020-10-01')
r.text[:10]
Out[7]: '{"from":16'

Dit lijkt vrij fool-proof te werken.

mediaminister commented 4 years ago

Thanks @canihavesomecoffee I made a pull request based on your solution.

frankdpGH commented 4 years ago

Thanks for the quick fix!!!

add-ons / plugin.video.vtm.go

TV Guide doesn't work anymore #209