raptor2101 / Mediathek

GNU General Public License v3.0
45 stars 16 forks source link

Arte #96

Closed ebora closed 5 years ago

ebora commented 7 years ago

Looks like Arte has redisigned their website again...

None of the menu entries in Mediathek work any more. "ARTE+7" results in empty list without error in log. "Sendungen von A-Z" results in this error:

19:55:01.317 T:2597245936   ERROR: EXCEPTION Thrown (PythonToCppException) : -->Python callback/script returned the following error<--
                                             - NOTE: IGNORING THIS CAN LEAD TO MEMORY LEAKS!
                                            Error Type: <type 'exceptions.AttributeError'>
                                            Error Contents: 'NoneType' object has no attribute 'group'
                                            Traceback (most recent call last):
                                              File "/home/osmc/.kodi/addons/plugin.video.mediathek/default.py", line 90, in <module>
                                                mediathek.buildMenu(path)
                                              File "/home/osmc/.kodi/addons/plugin.video.mediathek/mediathek/__init__.py", line 106, in buildMenu
                                                self.buildMenu(path,treeNode);
                                              File "/home/osmc/.kodi/addons/plugin.video.mediathek/mediathek/__init__.py", line 114, in buildMenu
                                                self.buildPageMenu(treeNode.link,len(treeNode.childNodes));
                                              File "/home/osmc/.kodi/addons/plugin.video.mediathek/mediathek/arte.py", line 102, in buildPageMenu
                                                self.showCluster();
                                              File "/home/osmc/.kodi/addons/plugin.video.mediathek/mediathek/arte.py", line 170, in showCluster
                                                content = BeautifulSoup(self.regex_cluster.search(pageContent).group(1),"html.parser");
                                            AttributeError: 'NoneType' object has no attribute 'group'
                                            -->End of Python script error report<--
raptor2101 commented 7 years ago

Confirm, I'll try to fix this till weekend...

raptor2101 commented 7 years ago

youp they changed their layout ... a single json source instead of mutliple ones. Within the next two days I'am unable to repsond to this, but than this should be fixable

raptor2101 commented 7 years ago

ok please check the current version an report back errors if some occure...

ebora commented 7 years ago

Thanks for the quick work!

"arte+7", "Sendungen von A-Z" and "Kategorien" function correctly. Though I have the impression that mediathek is slower than before, but I could be wrong. Maybe these warnings I see several times in the log have something to do with that?

20:04:04.218 T:2572080112 WARNING: COMXImageFile::GetCodingType progressive images not supported by decoder
20:04:04.293 T:2572080112 WARNING: Previous line repeats 1 times.
20:04:04.294 T:2572080112  NOTICE: LoadJpeg: unable to load https://static-cdn.arte.tv/resize/6H6mjulQMlt0yKypfr9kpEJwZeQ=/940x530/smart/apios/Img_data/20/071395-006-A_2024740.jpg

The only serious issue is with "Suche", it doesn't work:

20:06:48.884 T:2588857328   ERROR: EXCEPTION Thrown (PythonToCppException) : -->Python callback/script returned the following error<--
                                             - NOTE: IGNORING THIS CAN LEAD TO MEMORY LEAKS!
                                            Error Type: <type 'exceptions.AttributeError'>
                                            Error Contents: 'NoneType' object has no attribute 'group'
                                            Traceback (most recent call last):
                                              File "/home/osmc/.kodi/addons/plugin.video.mediathek/default.py", line 95, in <module>
                                                mediathek.searchVideo(searchText);
                                              File "/home/osmc/.kodi/addons/plugin.video.mediathek/mediathek/arte.py", line 79, in searchVideo
                                                content = self.searchContent.search(pageContent).group(1);
                                            AttributeError: 'NoneType' object has no attribute 'group'
                                            -->End of Python script error report<--
raptor2101 commented 7 years ago

I will fix the search this afternoon....

It's unclear to my, why LoadJpeg fails isn't, the link is correct. May-bee a HTTPS related error? Does this hit only isolated pictures or are all pictures unable to load?

Regarding to you "speed"-observation. You are right. The first MainPage-JSON object doesn't offer all information needed. So i switched back to crawl any certain Object-ID and request the needed information via JSON API (~1s per request times 20 till 40 Objects found per side).

ebora commented 7 years ago

It's unclear to my, why LoadJpeg fails isn't, the link is correct. May-bee a HTTPS related error? Does this hit only isolated pictures or are all pictures unable to load?

Only some pictures. They don't load instantly, the warning is given and then after about 2 seconds the picture appears finally. When I visit an entry again the responding picture is then shown directly without delay.

But if I don't wait the 2 seconds and scroll further down, mediathek hangs for a while without message in the log. If I then scroll up again all the pictures appear directly. I guess while hanging mediathek gets the missing pictures...

When I try this with ZDF, pictures are sometimes a bit slow, too, but only about 1 second max and without warn message. And mediathek doesn't hang when scrolling fast there.

Regarding to you "speed"-observation. You are right. The first MainPage-JSON object doesn't offer all information needed. So i switched back to crawl any certain Object-ID and request the needed information via JSON API

That means it's because of the design of Arte's website and it won't get any faster, correct?

raptor2101 commented 7 years ago

Regarding the whole Picture stuff

The plugin doesn't load the Pictures it only offers the links, but it offers everytimes the links for Pictures with the highes Qualitiy. My this is a bandwidth problem on the CDN side. I will limit the max size ...

That means it's because of the design of Arte's website and it won't get any faster, correct?

Yes and no - the plugin currently goes the "sledgehammer approach" (simply regex all possible IDs out of a side). I'am currently looking if all sides offer the same "baseline" of infos , so i can rely on them. Currently only the media link is missing, this could be loaded if you decide to "play" an entry. This would speed up the things.

Sorrily I left my workspace yesterday at 22:00 o'clock so i was unable to do further work on the plugin

raptor2101 commented 7 years ago

please test again ...

ebora commented 7 years ago

Loading doesn't appear to be faster. The only difference I see is that part of the pictures have less height now, they are about 4-5 times as broad as high. The warning message seems to be only with the old picture format.

Did the sorting change? I belive it was newest first before (which was good), now it is unsorted as far as I can see.

The search seems to work, only the first try didn't, but it is not reproduceable, the next 8 tries worked... Here is the error message for the first try anyway:

20:00:18.146 T:2653942768   ERROR: EXCEPTION Thrown (PythonToCppException) : -->Python callback/script returned the following error<--
                                             - NOTE: IGNORING THIS CAN LEAD TO MEMORY LEAKS!
                                            Error Type: <type 'exceptions.KeyError'>
                                            Error Contents: ('VSR',)
                                            Traceback (most recent call last):
                                              File "/home/osmc/.kodi/addons/plugin.video.mediathek/default.py", line 95, in <module>
                                                mediathek.searchVideo(searchText);
                                              File "/home/osmc/.kodi/addons/plugin.video.mediathek/mediathek/arte.py", line 81, in searchVideo
                                                self.extractVideoLinks(pageContent,0);
                                              File "/home/osmc/.kodi/addons/plugin.video.mediathek/mediathek/arte.py", line 259, in extractVideoLinks
                                                displayObject = self.extractVideoLinksFromJSONPage(jsonPage["videoJsonPlayer"])
                                              File "/home/osmc/.kodi/addons/plugin.video.mediathek/mediathek/arte.py", line 264, in extractVideoLinksFromJSONPage
                                                videoLinks = self.extractLinks (jsonPage);
                                              File "/home/osmc/.kodi/addons/plugin.video.mediathek/mediathek/arte.py", line 214, in extractLinks
                                                vsrObjects = jsonObject["VSR"];
                                            KeyError: ('VSR',)
                                            -->End of Python script error report<--
20:00:18.690 T:3025780736   ERROR: GetDirectory - Error getting plugin://plugin.video.mediathek/?action=search&type=ARTE
20:00:18.873 T:3025780736   ERROR: CGUIMediaWindow::GetDirectory(plugin://plugin.video.mediathek/?action=search&type=ARTE) failed
raptor2101 commented 7 years ago

the sorting comes directly from arte, within a section sorted by date i guess.

I will fixe the search

regarding to speed, this seams strange cause currently only one page is loaded instead of 40...

raptor2101 commented 7 years ago

by the way the current version is pushed to upstream

ebora commented 7 years ago

I have been busy with other things but am pretty sure that version 0.8.3 worked initially. But now I get the following error every time I want to play a video:

19:30:08.131 T:3005215728   ERROR: EXCEPTION Thrown (PythonToCppException) : -->Python callback/script returned the following error<--
                                             - NOTE: IGNORING THIS CAN LEAD TO MEMORY LEAKS!
                                            Error Type: <type 'exceptions.IndexError'>
                                            Error Contents: list index out of range
                                            Traceback (most recent call last):
                                              File "/home/osmc/.kodi/addons/plugin.video.mediathek/default.py", line 104, in <module>
                                                mediathek.playVideoFromJsonLink(link);
                                              File "/home/osmc/.kodi/addons/plugin.video.mediathek/mediathek/arte.py", line 210, in playVideoFromJsonLink
                                                self.gui.play(links);
                                              File "/home/osmc/.kodi/addons/plugin.video.mediathek/simplexbmc.py", line 195, in play
                                                link = self.extractLink(links);
                                              File "/home/osmc/.kodi/addons/plugin.video.mediathek/simplexbmc.py", line 210, in extractLink
                                                selectedKey = links.keys()[0];
                                            IndexError: list index out of range
                                            -->End of Python script error report<--
19:30:08.668 T:3025698816   ERROR: Playlist Player: skipping unplayable item: 0, path [plugin://plugin.video.mediathek/?type=ARTE&action=openJsonLink&link=https%3A%2F%2Fapi.arte.tv%2Fapi%2Fplayer%2Fv1%2Fconfig%2Fde%2F071395-001-A]

Unfortunately I won't be able to do any testing in the next 2 weeks...

raptor2101 commented 7 years ago

please retest

ebora commented 7 years ago

works again, thank you!

The slow scrolling issue persists, though, but I can live with that for now. Maybe we should make a separate ticket for that?

And I noticed a minor formatting thing: when I go to "Sendungen A-Z" all the entries (which are folders, not files, e.g. 360°, Abgedreht) have a date in front of them which doesn't make sense in that context (it isn't even the date of the last new instance of that show). But when I select one of these show-folders the entries there (which are files/actual broadcasts) don't show a date. But with this files I would like to know that date (and maybe have sorting by that date, too). If I select "Suche" in the main menu, the result is shown as expected and desired: I search for e.g. 360° and get a list of all files of that show with dates.

Auerhahn commented 6 years ago

I just installed the Mediathek addon. It works fine with ARD and some other stations, but it doesn’t with arte and ORF. I get the message »Error« and shall look to the log file. I don’t know where to find the log file... As I see here, this is an old issue. Is there a chance to get this fixed?

raptor2101 commented 6 years ago

I'm currently working on it patches incomming...

raptor2101 commented 6 years ago

they added one more layer of abstraction to the initial json... with some testing i'am able to push a patch tomorow...

raptor2101 commented 6 years ago

can someone retest arte?

digitalmind commented 6 years ago

Any updates?

raptor2101 commented 6 years ago

update is pushed to addon-repo, waiting for feedback, works on my HTPC...

digitalmind commented 6 years ago

Sadly it does not work on my vero

Traceback (most recent call last): File "/home/osmc/.kodi/addons/plugin.video.mediathek/default.py", line 104, in <module> mediathek.playVideoFromJsonLink(link); File "/home/osmc/.kodi/addons/plugin.video.mediathek/mediathek/arte.py", line 224, in playVideoFromJsonLink jsonObject = json.loads(self.loadPage(link)); File "/home/osmc/.kodi/addons/plugin.video.mediathek/mediathek/__init__.py", line 69, in loadPage return content.text.encode(content.encoding); TypeError: encode() argument 1 must be string, not None -->End of Python script error report<-- 19:46:00.804 T:4076360272 ERROR: Playlist Player: skipping unplayable item: 0, path [plugin://plugin.video.mediathek/?type=ARTE&action=openJsonLink&link=https%3A%2F%2Fapi.arte.tv%2Fapi%2Fplayer%2Fv1%2Fconfig%2Fde%2F081858-002-A%3FlifeCycle%3D1]

raptor2101 commented 6 years ago

is this problem persistent on you device i'am able to open the link