search

raptor2101 / Mediathek

GNU General Public License v3.0
45 stars 16 forks source link

Make Mediathek Addon capable to fetch MediathekView Filmliste [fixed by independent plugin https://github.com/YggdrasiI/plugin.video.simple_mediathek_de] #67

Closed Wikinaut closed 7 years ago

Wikinaut commented 8 years ago

UPDATE 20170419: This idea is now implemented in an independent plugin: https://github.com/YggdrasiI/plugin.video.simple_mediathek_de

I suggest to add a new "pseudo station" to Mediathek AddOn which fetches the Filmliste of MediathekView and allows to search that list, and to select movies for viewing and downloading.

See also https://github.com/raptor2101/Mediathek/issues/55#issuecomment-174413573 and https://github.com/xaverW/MediathekView/issues/60 (corresponding issue on MediathekView "[Suggestion] Add MediathekView as Addon for OSMC/KODI").

Here's an edited excerpt of the Filmliste:

{
  "Filmliste" : [ "25.01.2016, 16:19", "25.01.2016, 15:19", "3", "MSearch [Rel: 516] - Compiled: 21.01.2016 / 18:28:22", "874e30dca266b1fe704bc4cb728f1c66" ],
  "Filmliste" : [ "Sender", "Thema", "Titel", "Datum", "Zeit", "Dauer", "Größe [MB]", "Beschreibung", "Url", "Website", "Untertitel", "UrlRTMP", "Url_Klein", "UrlRTMP_Klein", "Url_HD", "UrlRTMP_HD", "DatumL", "Url_History", "Geo", "neu" ],
  "X" : [ "3Sat", "3sat", "Eisige Welten (3/5)", "25.01.2016", "21:40:00", "00:43:06", "762", "Es ist Sommer. Ein einziger Tag erstreckt sich über Wochen, bis zu sechs Monaten! Kaum vorstellbar, aber die Sonne geht nicht mehr unter. Im gleißenden Licht des Polartages schmilzt den Eisbären...", "http://nrodl.zdf.de/dach/3sat/15/01/150118_eisige_welten3_online_2256k_p14v11.mp4", "http://www.3sat.de/mediathek/?mode=play&obj=48754", "", "", "65|436k_p9v11.mp4", "", "", "", "1453754400", "", "DE-AT-CH", "true" ],
  "X" : [ "", "", "Population Boom © ORF/www.austrianfilm.at/Geyrhalter Film", "24.01.2016", "21:45:00", "01:29:11", "1563", "Der preisgekrönte österreichische Regisseur Werner Boote (\"Plastic Planet\") bereist für seinen Dokumentarfilm unsere Erde, die mit sieben Milliarden Menschen bereits überbevölkert ist....", "http://nrodl.zdf.de/dach/3sat/16/01/160124_population_boom_online_2328k_p35v12.mp4", "http://www.3sat.de/mediathek/?mode=play&obj=56521", "", "", "66|436k_p9v12.mp4", "", "66|3328k_p36v12.mp4", "", "1453668300", "", "DE-AT-CH", "false" ],
  "X" : [ "", "", "Eisige Welten (2/5)", "25.01.2016", "21:00:00", "00:43:09", "763", "Beginnend mit der großen Schmelze im Frühling steht die zweite Folge ganz im Licht der zurückkehrenden Sonne. Ihre Wärme verwandelt die magische Eiswelt nach einem halben Jahr Dunkelheit. Noch...", "http://nrodl.zdf.de/dach/3sat/15/01/150111_eisige_welten2_online_2256k_p14v11.mp4", "http://www.3sat.de/mediathek/?mode=play&obj=48700", "", "", "65|436k_p9v11.mp4", "", "", "", "1453752000", "", "DE-AT-CH", "true" ],
  "X" : [ "", "", "Nuhr im Ersten © RBB", "24.01.2016", "21:00:00", "00:45:27", "796", "Scharfzüngig und anspruchsvoll, bissig und hintergründig, unterhaltsam und witzig - Kabarett und Comedy treffen sich bei \"Nuhr im Ersten - Der Satire Gipfel\".", "http://nrodl.zdf.de/none/3sat/16/01/160124_nuhrimersten_ard_2328k_p35v12.mp4", "http://www.3sat.de/mediathek/?mode=play&obj=56550", "", "", "60|436k_p9v12.mp4", "", "60|3328k_p36v12.mp4", "", "1453665600", "", "", "false" ],
  "X" : [ "", "", "Eisige Welten (1/5)", "25.01.2016", "20:15:00", "00:43:09", "762", "Die erste Folge entführt in Arktis und Antarktis und stellt sie als Eis gewordene Superlative vor. Fast 80 Prozent des Süßwassers unseres Planeten sind dort gebunden, Winde peitschen mit 320...", "http://nrodl.zdf.de/dach/3sat/15/01/150111_eisige_welten1_online_2256k_p14v11.mp4", "http://www.3sat.de/mediathek/?mode=play&obj=48699", "", "", "65|436k_p9v11.mp4", "", "", "", "1453749300", "", "DE-AT-CH", "true" ],
    ....
  "X" : [ "", "", "Ausschnitt: Marinas Rettungsmission", "21.10.2015", "16:00:00", "00:02:15", "40", "Marina will die Schildkröteneier in Sicherheit bringen. Dabei trifft sie auf die Gauner Rico und Biff. Was haben die beiden nur vor?", "http://nrodl.zdf.de/none/tivi/15/10/151014_folge2_ausschnitt_zoo_2256k_p14v11.mp4", "http://www.tivi.de/fernsehen/zoom-der-weisse-delfin/start/index.html ", "", "", "", "", "", "", "1445436000", "", "", "false" ]
}
raptor2101 commented 8 years ago

as written in #63 downloading things won't be implemented soon but "scapping" the "Filmliste" for displaying seems interesting.

Is the link static ? or does it changes from release to release?

Wikinaut commented 8 years ago

@raptor2101 wrote

Is the link static ? or does it changes from release to release?

I don't know. Perhaps @xaverW can answer ?

Wikinaut commented 8 years ago

crossposted: @raptor2101 the youtube-dl Addon can download in the background!

This is the source:

Description: http://kodi.wiki/view/Add-on:Youtube-dl_Control

It takes arbitrary stream urls, not only Youtube-Urls. I use this already to download "Tatort" or so.

raptor2101 commented 8 years ago

ask em ;)

Wikinaut commented 8 years ago

@raptor2101 I asked him.

xaverW commented 8 years ago

whitch URL?

Wikinaut commented 8 years ago

Url der "Filmliste" (json).

@raptor2101 The urls of servers holding the Filmliste appears to be in http://zdfmediathk.sourceforge.net/akt.xml .

xaverW commented 8 years ago

planed is nothing BUT: this is for the program and when it needs changes, then I would change it.

raptor2101 commented 8 years ago

@xaverW As long as you have no problem that third party apps uses this file everything is fine ... if you change the link I will adoptthe change. My purpose of the question was to prevent a "cat an mice" game ;)

Wikinaut commented 8 years ago

@raptor2101 Problem with your Mediathek App ? I was looking for "Geierabend 2016" which has http://www1.wdr.de/mediathek/video/sendungen/comedy/videogeierabend102.html in the WDR Mediathek, and found, that WDR Mediathek is not part of your Mediathek App.

Why is WDR Mediathek not listed ?

Wikinaut commented 8 years ago

I would really prefer you make all available video of the "Filmliste" (as mentioned in my initial post above https://github.com/raptor2101/Mediathek/issues/67#issue-128563023 ) searchable in your Mediathek App, and then playable.

Wikinaut commented 8 years ago

@raptor2101 or use his "feed" list http://linkeddata.mro.name/open/tv/mediathek/ardmediathek.de/series/index.opml

raptor2101 commented 8 years ago

So many questions ... I skipped WDR support a while ago. At this WDR changes their front-end very frequently and uses ways to "protect" their content in a way, that make it perfectly clear, they don't want to be crawled by a third party tool. Additional most content was available within the ARD mediathek, so i focused on this part... A fragment of the original scrapper is still shipped with the addon but disabled, used at your own risk.

Additional I looked how to implement your request. Thinks get very suspicious... if I get it right these opml.file (mirrored ...) only overs a direct link to the series-pages hosted by ARD. Once get this page, the video-link has to be extracted as already implemented in the ard.py file.

Some random samples shows that most of the content is already scrapped by the ard.py. what is exactly the content found in this opml wich is not scrapped by the ard-plugin?

maybe a small change in the scrapper solves these problem....

Wikinaut commented 8 years ago

@raptor2101 tja, basically I thought, that it might be quite easy to "convert" presented texts (titles) and urls (stream links) into a searchable and clickable list in your add-on. Once I have learned how this can be done, I certainly can help to fetch (to include) more links, e.g. the "Filmliste" of MediathekView.

Wikinaut commented 8 years ago

@raptor2101 let me know, how I can help with a bitcoin donation to you.

raptor2101 commented 8 years ago

It's not about money. I'am doing this for fun ...

If i get you right, you are missing the search functionality? Using the "Filmliste" doesn't help here, because to search something Kodi has to load all linked source-sides and do a brute force search.

Using ARD base search for this looks more promising ... I will look into this this week ... if my time isn't consumed by my family or xcom2 ... :D

Wikinaut commented 8 years ago

@raptor2101 why do you want to load all linked source-sides ? The Filmliste has all the meta information, e.g.

"X" : [ "", "", "Nuhr im Ersten © RBB", "24.01.2016", "21:00:00", "00:45:27", "796", "Scharfzüngig und anspruchsvoll, bissig und hintergründig, unterhaltsam und witzig - Kabarett und Comedy treffen sich bei \"Nuhr im Ersten - Der Satire Gipfel\".", "http://nrodl.zdf.de/none/3sat/16/01/160124_nuhrimersten_ard_2328k_p35v12.mp4", "http://www.3sat.de/mediathek/?mode=play&obj=56550", "", "", "60|436k_p9v12.mp4", "", "60|3328k_p36v12.mp4", "", "1453665600", "", "", "false" ],

Use grep or my agrep (for fuzzy searching) the title in the json object. E.g. when the user entered "Nuhr" the example above is one of the results. Then present the user the available formats, and "fire" the playing!

Wikinaut commented 8 years ago

@raptor2101 wrote

Using the "Filmliste" doesn't help here, because to search something Kodi has to load all linked source-sides and do a brute force search.

I don't know the details of KODI, but what I suggest is to write a script (new addon, nom de guerre: "Filmliste Addon") which does:

Where exactly is the problem ?

raptor2101 commented 8 years ago

Ok now i,am a little confused. The direct link you posted contains an xml leading to an index of the whole Ard mediathek.

Where you get the json from?

Wikinaut commented 8 years ago

This script https://github.com/Skeeve/Mediathek-3.3-Filmliste/blob/master/filmliste_laden explains it all.

First, fetch the "server list" (list of mirrors of the current filmliste). Then try to fetch (by random selection) one of the mirrors, until you can download one copy of the Filmlist.

Then extract...

raptor2101 commented 8 years ago

Sometimes I'am blind ...

If I'am getting it right, this is a "full" index of all videos currently available ...

Loading the file, uncompress em within python (without external tools) will be the first task.

Parsing the file into chunks, usable and searchable (from within Kodi) is the interesting part. the current mediathek addon isn't prepared for that kind of "pre-processing/caching" so I will create a new Addon

I will post a preview as soon as I have something to show ...

Wikinaut commented 8 years ago

@raptor2101 wrote

Sometimes I'am blind ... If I'am getting it right, this is a "full" index of all videos currently available ...

I am happy to see that you've got it, that it looks relatively easy to code. (thanks to @xaverW , the maintainer and main coder of the great MediathekView, whose "Filmliste" is the big "thing" here, and perhaps of big help.)

Wikinaut commented 7 years ago

@raptor2101 Now, where MediathekView has a new team (see https://mediathekview.de/ ), I want to kindly ask whether we can resume our discussion, and perhaps coding, to write a "new" MediathekView Video Addon, which - as described above - makes use of the prefetched list of available recordings (today: more than 172.000).

As a starting point, a simple full text search function would be fine as basic interface, which then needs to "simply" play the attached stream url (perhaps with an additional selection of the stream quality HD, HQ or SD).

Wikinaut commented 7 years ago

NEW: https://github.com/bagbag/mediathekviewweb is a new web interface which access the whole mediathekview database. Can be used as a blueprint to develop a similar KODI AddOn, as requested in this issue.

raptor2101 commented 7 years ago

will look into this as soon as i have time. i'am currently heavily consumed by my job. will report back as soon as possible

Wikinaut commented 7 years ago
  1. download http://zdfmediathk.sourceforge.net/akt.xml
  2. random-pick one of the mirrored copies, and download, uncompress it (currently, the uncompressed filesize is about 115 MByte)
  3. load the json data (load only the fields, which are needed for searching) and make it somehow searchable similar to https://mediathekviewweb.de/ (source code https://github.com/mediathekview/mediathekviewweb )
  4. after search, when the App user selected one search result for playing, retrieve the stream url in the selected resolution from the json file, or allow downloading that file for later viewing.
Wikinaut commented 7 years ago

Implementation under progress here:

raptor2101 commented 7 years ago

So i can close this one

Wikinaut commented 7 years ago

@raptor2101 hm, yes. When filing this issue, I wanted (wished) you were adapting your plugin to use the filmliste, but the new work - I haven't used it yet - appears to do this. So closing is fine for now.

raptor2101 commented 7 years ago

@Wikinaut as you might notice, my response-time is rather slow ... since one year, i'am really occupied by my job. Because this is heavily related to "coding", coding at home isn't the right method to get the head clear. As soon as my Addon-functionality is fully covered by the other Addon, I will close this one ...