Closed iz8mbw closed 6 years ago
Here is the log file if you would be willing to help me to get it working. http://paste.ubuntu.com/26204288/
Please?
@Sandmann79 any news on enabling Italy as country?
I'm currently working on it (just a friendly stranger hacking around). So far I've got the login working for primevideo.com, and I'm working on the live scraping of series (well, more like working on the digestion of information I can already scrape). It should be possible to add the series images as well along the titles and possibly other information, but it's the first time I'm hacking a kodi addon, eventually someone will hopefully take the enhancing part forward when the main part is done (as in: proper login and videos actually working in Italy and elsewhere).
In regard to the location selection, I've noticed there seems to be 3 or 4 area codes, so it will probably need another configuration option if the selected provider is primevideo.com, but it doesn't seem too strictly related to the content providing. I think once I fix this bit, since the playlists are (apparently) provided through geolocalization, the italian content will be provided in Italy while the indian content will be provided in India, there's not much configuration to do past that. In addition to that we won't have to rely on json menus provided statically on github or anywhere else. It will probably lengthen the initial loading times, but I think it's an acceptable compromise.
So far full support seems possible/plausible, but as I said I'm still working on the scraping part and re-engineering/re-factorization of some bits of the addon code to enable the usage of primevideo.com. Hopefully I will have something working in a few days, if my job doesn't get in the way too much.
PS: @oomek I'm not 100% sure, but the list of movies probably don't work because they're tied to the json files on the github repo, and they haven't been updated in months. Hopefully the primevideo.com addition will fix this problem too.
PPS: probably nobody is interested in this little fact, but I noticed Amazon doesn't use JSON, it uses json-amazonui-streaming. kk, then :|
PPPS: disregard the part about loading times, I just realized that we need to load stuff on request, except maybe for the movie and series categories list. With those many series and multiple page lists, it would be an excessive strain put on their servers, and useless wasted loading time on the user end. Also amazon seems to return wrong series lists during pagination, will have to find out how to avoid that too.
So they have had prime out for Canada for a bit now. They recently released the fire tv stick basic in Canada. If you buy it you can get prime. Or if you use browser. Or if you use app on any other android device. However I have a fire tv box, gen1 stick, gen2 stick all purchased in USA and none support prime outside USA (stupid) so I too would love getting this addon working in Canada. Thanks.
@drizzt09 something could (theoretically) be done about that, but I could need the help of canadians. I don't remember seeing in the code anything specific about amazon.ca, do you also go to primevideo.com? If so when I finish this little hacking/reverse engineer adventure, it should probably work for you too.
@varstahl yes. Also use https://www.primevideo.com/ref=av_auth_return_redir I will test for you but I only have 9 days left on my free trial
Ok then, it would probably work alright. I don't know if I can make it in time though, I'm hitting several obstacles one after another atm. It seems like primevideo.com just uses /search/
for category listing, and search has no easily parsable JSON format available, so I'm stuck to JSON parsing for something and HTML parsing for the rest. I'll make it work, eventually, I'm as interested as anybody else, but I'm wondering if there's some hidden API hiding somewhere that could make our life better (and speed things up dramatically).
There is another kodi addon in the suprrepo. I haven't tried it but it also doesn't work with other countries. But code may help
Since I've started the series scraping/parsing I've ignored the addon code, I'm just straight up reverse engineering the website old-school, and updating a tiny python script in preparation. I've been working on it for a couple days and got most the information and procedures needed going pretty smooth, I just fear I might hit a brick wall soon, whether due to technical issues (unlikely) or due to design issues (more likely).
One example would be the movie/series synopsis: do I perform an additional query for the additional information we can gather from the movie page, and provide the video straight from the movie category list? Do I add a "virtual folder" (for lack of better word) and once going in it in Kodi provide the movie video link to stream, along with the virtual information (this would be necessary for the series anyway)? Or do I completely ignore the synopsis and go for the least clicks possible?
Another example: currently for one movie category search it bears ~80 movies worth of results, in a pagination of 20 results per page. Do I scrape all of them right away or do I insert a pagination in Kodi to reflect the pagination on the primevideo.com? The problem is that inserting a pagination in Kodi would kind of be a problem I'm not sure how to solve atm, while scraping them right away might mean going through hundreds to parse and then sift through in Kodi.
To paraphrase O'Neill "I'll just have to cross that bridge when I come to it".
Thanks Bruno for the amazing effort you're putting into this! Cheers,
-- Andrea
2018-01-17 3:09 GMT+01:00 Bruno Passeri notifications@github.com:
Since I've started the series scraping/parsing I've ignored the addon code, I'm just straight up reverse engineering the website old-school, and updating a tiny python script in preparation. I've been working on it for a couple days and got most the information and procedures needed going pretty smooth, I just fear I might hit a brick wall soon, whether due to technical issues (unlikely) or due to design issues (more likely).
One example would be the movie/series synopsis: do I perform an additional query for the additional information we can gather from the movie page, and provide the video straight from the movie category list? Do I add a "virtual folder" (for lack of better word) and once going in it in Kodi provide the movie video link to stream, along with the virtual information (this would be necessary for the series anyway)? Or do I completely ignore the synopsis and go for the least clicks possible?
Another example: currently for one movie category search it bears ~80 movies worth of results, in a pagination of 20 results per page. Do I scrape all of them right away or do I insert a pagination in Kodi to reflect the pagination on the primevideo.com? The problem is that inserting a pagination in Kodi would kind of be a problem I'm not sure how to solve atm, while scraping them right away might mean going through hundreds to parse and then sift through in Kodi.
To paraphrase O'Neill "I'll just have to cross that bridge when I come to it".
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Sandmann79/xbmc/issues/22#issuecomment-358171890, or mute the thread https://github.com/notifications/unsubscribe-auth/ADgAL-seKMFO1bpDqboB06rzAUlo8mvKks5tLVZmgaJpZM4LM-WY .
@Varstahl, first off all thank you for your work! Having Amazon Prime Video working on Kodi for the different countries will be just amazing! And I understand the effort and passion you are putting on this project (I have also developed a Kodi add-on and it was an amazing experience...luckily it seems I have reached a stable version that do not need further enhanecements at the moment). Ok, after this long introduction my two cents on your points above...
About the synopsis it would be great to have it already scraped when scrolling the movies/series lists to have an idea about what the movie/series is proposing. There are currently some Kodi views that show the synopsis so it would be a pity to have that field empty and only available if you go into the movie/series information page.
Regarding the pagination I would go for the complete scrape of the movie/series list. May be this will take some times but I don't think is going to be an issue for the final user. I guess that Kodi will display by default the first 20/25 items and then add a
Anyway, I would suggest to give a look to this add-on called TIMVISION; Amazon Prime Video and TIMVISION have quite similar site structure so that you can take a look to that add-on for some inspiration. May be you can also contact the author to exchange some views.
With that...good work and thanks again!!!!! ;)
@axlt2002 I'm also "sort of" of the same opinion in regard to the synopsis, but my problem is here, for example some movies like Elysium have a truncated synopsis which is only available in full in the full movie page (Either pv bugged for me or I was so tired last night I didn't even notice that the synopsis is truncated even in the full page). Scraping that would building up 100+ heavy queries the very single moment you try to look at a movie list. Unless I find an API endpoint to work around that, I guess I'll keep the truncated version of the synopsis in the list and the full synopsis on the movie/series folder. Same goes for the full page background image.
Although you technically only load it once when the section loads for the first time, I'm not sitting comfortably running a not well defined number of queries in the foreground (or the background for that matter).
As for the pagination, yeah, although it can build up quickly it's just (at least atm) 4-5 pages per category, it's not too bad scraping them right away.
Cool, I was checking at the example you posted and noticed that even in the full page the synopsis is truncated...and then I saw that you modified your post too! That sounds weird anyway...it seems that there is no way to retrieve the full plot...
Despite truncated or not, the TIMVISION site has the same outline of Amazon Prime Video and the TIMVISION add-on scrapes the plot and the fanart (i.e. background image) from the movie detail page to use them in the list view; I can guarantee that the user experience is not impacted at all despite the number of queries that are performed.
I'll take a look at it later, I'm currently parsing/building series data. The optimal thing would be to simulate a hover, so the basic information is loaded immediately from the listings and the background images and such get loaded when the user moves over the movie title. I don't even know if that's possible in kodi, but again I'm still working on the backend.
If someone wants to test the changes around, feel free to download the current WIP from my fork. As of now there's not much functionality available, but the API & video endpoints are correct, the login is working, and the categories load (partially) on the fly. I just found out that the addon works pretty much like a server script, so my plan to have a per-session structure kind of folded on me atm.
Anyway there's a new option in the menu: in order to make it work you have to select "Other (primevideo.com)" from the country list, the area codes then unlock. There are currently four areas, ROE_EU, ROW_EU, ROW_FE and ROW_NA, though I'm not quite clear on their use (Rest of European Union / EUROPE, Rest of World / Europe, Rest of World / Far East, Rest of World / North America? Maybe?). Italy has to use ROE_EU, India and Japan or the nearby areas ROW_FE, Canada ROW_NA, the rest I have no idea.
Give it a shot, you should see Movies, TV Shows and Kids root categories, and when you step into them you should see a number of categories in your locale, matching the results from primevideo.com.
There are still things to fix here and there, but it's late af and gotta catch some sleep :-]
@Varstahl i dont see a download zip option on your fork
How do I get the plugin? Not working at all. :(
@Varstahl can you enable the "issue" funcionality in yout github repo? I think that will be more easy to track if we discuss specific topic in new issues instead use this as a open fourm. @drizzt09 It's a alpha version not ready for standard users, you must compile it.
Hi @Varstahl thanks for doing what i don't know how ;)
I'm from Spain (ROE_EU), and I want to help debugging your fork, I'll download. If you need any kind of log, please comment it and i'll do my best to get it.
Thanks Again
@drizzt09 @theboy181 at the moment there is no zip option (intended as addon-ready zip) because I'm just developing and updating before merging with @Sandmann79 repository. It won't let you see videos yet, but if you want to play around with it you can either download the entire repo zip from here or manually download the files from the plugin.video.amazon-test directory. Once you have the plugin.video.amazon-test files, you can unpack them inside %AppData%\Kodi\addons\plugin.video.amazon-test
in windows or ~/.kodi/addons/plugin.video.amazon-test
in linux. I repeat though: it's still not entirely working, it's a work in progress.
@CGarces yeah I will, I completely forgot to last night, I'll enable it right away.
Personally I have created a plugin.video.amazon-test.zip and installed it directly from Kodi. Just to notify that in such a way the dependency related to script.module.pydes-2.0.1 was not able to be met, so that I had to manually download and install it separately. May be this could be solved in future by creating a repository (I guess that such a module is not available in the official Kodi repository).
I will try the test version as soon as possible.
Thanks again for your effort Varstahl!
Anoter bit on my side...my antivirus Symantec Endpoint Protection dected a Trojan inside the file userinput.exe under the folder tools of the add-on. I'm sure is a false positive but I wanted just to highlight this.
@axlt2002 it actually is, I just installed the old plugin.video.amazon-test from @Sandmann79 repo and then edited on top of it. Strange. Also for the userinput.exe that's part of the original package, I have no idea what that does (possibly remote control helper?), but as he said, being that it uses AHK scripts it's quite normal.
No problem, I will try with a clean installation to check if the problem persists. I will report back to you in case.
Hi @Varstahl I am trying to help here installing the in-progress version of this add-on but I did not succeed :(
I basically followed your above steps (copy 'plugin.video.amazon-test' files into '~/.kodi/addons/plugin.video.amazon-test'), but when I launch kodi I don't see the add-on on the UI, hence I cannot use it. Do I need to do something else?
Thanks for your great job!
Diego
@DiegoAntonino the addon needs to be also enabled when put in there manually, you should find it in the list of available addons (it should list both enabled and disabled addons). You might run into dependency issues though if you lack a few them upon which Amazon VOD
is based. I've cooked up a one-page wiki where you can find the general information on the changes I'm bringing in, you can find a short how-to install right here.
TL;DR: install Sandmann's original addon and copy my files over them.
Got it. Thanks!
I was able to login (somehow it took a while to do so). When I login and set 'ROW_NA' as a PrimeVideo Area (since I am in Canada) I have just 3 options [movies, kids and TV Shows] and I was not able to watch anything.
Movies -> it shows several categories and when I select one of them I got the following error: `13:58:03.957 T:1797255936 NOTICE: [Amazon VOD] {'path': 'root-//-Movies', 'mode': 'PV_Catalog'} 13:58:03.975 T:1797255936 NOTICE: [Amazon VOD] getURL: https://www.primevideo.com/storefront/movie?_encoding=UTF8&format=json 13:58:06.386 T:1797255936 NOTICE: [Amazon VOD] getURL: https://www.primevideo.com/storefront/movie?_encoding=UTF8&node=&page=2&pageSize=14&total=19&format=json 13:58:12.004 T:1797255936 NOTICE: [Amazon VOD] {'path': 'root-//-Movies-//-Comedy Movies', 'mode': 'PV_Catalog'} 13:58:12.010 T:1797255936 ERROR: EXCEPTION Thrown (PythonToCppException) : -->Python callback/script returned the following error<--
Kids -> shows empty with the following log:
13:58:40.529 T:1453323008 NOTICE: [Amazon VOD] {'path': 'root-//-Kids', 'mode': 'PV_Catalog'} 13:58:40.547 T:1453323008 NOTICE: [Amazon VOD] getURL: https://www.primevideo.com/storefront/kids?_encoding=UTF8&format=json
This is because there is not a kids section :)
I see PV_Catalog is failing because cannot find "Comedy Movies" in pvCatalog.... But I am not sure what this part should do:
for n in path.split('-//-'): node = node[n] if 'lazyLoadURL' in node: PV_LazyLoad(node)
The categories don't work yet because I didn't have a caching system, being the first time I develop for Kodi I didn't know how some parts were managed. It's currently partially fixed in my offline codebase, I will upload it as soon as I can, I'm fixing some unicode problems.
Anyway I'd invite you guys to post this kind of stuff in the issue section of my fork, so as not to spam all the other people with it, and so that I can manage the eventual problems better.
TL;DR: the behavior is as expected so far (should probably upload a fix tonight), also glad it works in Canada :) Also: at this stage no, you won't be able to watch anything anyway, you could see the movie/series titles and metadata at best. I'm slipping fixes and adjustments one by one as soon as I can, but real job comes first ;)
First of all, big thanks @Varstahl to take care of the problem. Have a look at this Amazon addon: https://github.com/liberty-developer/plugin.video.prime_instant. Maybe this helps or is better as base, because it's scrapps the Amazon webpage, similar to your changes. @axlt2002 this is an false positive. More info at this issue #61
Took a look at the plugin, but as far as I can tell it's of no use. Before I started working directly into the addon, I scraped the data roughly from the JSON-AmazonUI-Streaming, and in part from the raw HTML (for the seasons episodes, mainly). I then started thinking "well, if I could batch out 40+ ASIN requests with the ASINList it could provide better information and speed things up", turns out most of the AJAX API endpoints have been nulled and removed. GetASINDetails
just won't work with PrimeVideo, it always returns 0 titles, and after most of yesterday spent around it, trying to figure out whether I messed something up or not, I came across a post from a few months back where amazon clearly stated in the AJAX Response "This is not working anymore and never will, ever again". Apparently they transitioned from a hybrid solution to a strictly HTML5 approach, and they restructured the API and workflow in doing so.
Sooo… I'm back to the original scraping of mixed faux JSON and HTML. Got most of the information and a JS endpoint to get the video resources back, now I "just" have to make it work :-]
Edit: I just dived in the metadata/art kodi management business, and it came out pretty ok. Added a bit of new information on top of the work I already done before inserting it into the addon, and they seem to be working fine. Most of the series informations, thumbnails, plot, actors, rating, fan art and so on is working, though some work is still to be done on the scraping. Ah yeah, that and actually playing videos, that'd help too :D
For the most impatient ones, I got primevideo to work (tested on Win7 & Debian Linux on Kodi 18a). You can find instructions on how to manually install my update here. There are still a number of QoL improvements to implement, but at least now the entire catalog is playable.
As for the pic or didn't happen (pardon the quality):
In the meantime while hitting a wall head first before discovering 32 hours later that I had everything working perfectly except telling Kodi that it actually had to start the video player, I noticed that there was no "Remember Me" when logging in. If you relog now (with my update installed) you should be able to stay logged in for a long time now, instead of 12/24 hours. I will introduce an option to do that, instead of doing it by default, in the near future.
@Varstahl i would like to test this process, I already have sandmans zip, repo, addon installed in Kodi (Android FTV). You mentioned earlier that yours doesnt have zip and I must compile... I do not see a "compile" option to do this. I can easily copy all your files over to where they need to be in Android ...all at once, but i dont know how to get them all at once from your GIT. thank you
@drizzt09 no, there's no compile of sorts, that was someone else mistake. You can get the entire zip from here, just make sure you unpack the plugin.video.amazon-test
only. You might need to also remove the resources/language
from the existing installation (just the first time). The language part is a bit of a bother, but as I was working on localization I discovered that the resource format used from the addon was obsolete) and so I just upgraded it.
OK got the zip, extracted the test folder only, deleted languages from original. replaced your files and restarted kodi. dont see canada in country and language is hardcoded to GER, cant change it now im getting error during playback that i need Kodi18... is that actually a requirement? I have 17.6
I have no idea about the Kodi version to be honest. I've got two 18 alpha and that's what I developed and tested on, I've read some things that said that it had improvements for InputStream.Adaptive, which is what we're using to play content from primevideo.com and netflix. I didn't code that part myself, so yes, it's possible that if you're using adaptive inputstream you might need kodi 18 for it to work.
As for the language are you sure you selected Other (primevideo.com) as seen in this screenshot?
i dont have "other" and I did have it on input stream, not sure the difference between script, browser and android which are the other options
There might have been a mistake then. Try to copy the files again, mainly these:
resources/language/resource.language.en_gb/strings.po
resources/settings.xml
default.py
If you can't see Other (primevideo.com)
then the resources/settings.xml
is the wrong one.
Hi Varstahl, In order to avoid such kind of installation issues, would not be more less time consuming for you to create a .zip package to be installed without the need of the previous version? The only issue would be the dependency I have highlighted before but it can be solved by installing the repository from Sandman79.
Of course this will be just a temporary solution. I guess that the two add-ons will be merged at certain point in time, right?
The problem is that, while technically correct, it would cause dependency problems requiring to install the addon from the repo anyway, and that the release I would pack would always be behind. I'm uploading a lot of commits daily, and there's no automatic way to do it in github, otherwise I'd already done it.
The most practical solution would be to just automate the thing with a script, pulling from my repo and copying the files over, but I'm lacking the time to develop a proper script, also given that (hopefully) in a few days it'll be done and over.
Edit: also, it's not really ready for public usage. I just discovered that InputStream.Adaptive ignores every non-English audio track. Kinda pissed on my parade…
Small update: with today's updates most of the features have been implemented, although we're far from being done. The root categories as reported by @Goruti has been fixed, I implemented a few adjustments as reported by @CGarces, there are now the "Remember Me" option during login, the custom views are enabled as per Amazon VOD's settings, fanarts for movies are now gathered upon scanning, and tvseries should now be correctly nested (when amazon doesn't screw up, like in the current state of the italian catalog).
The downside of the last two points are a severely longer scraping time (on some of the 40 pages sections we're talking minutes worth), on the upside the art and runtimes are correct for each movie and episode. So, there's that.
The english speakers could probably use this version for a good while, all the others should probably wait… I'm still unclear if the multilanguage stream problem is due to the addon, the InputStream.Adaptive, or Kodi itself. I'm still investigating the matter.
The downside of the last two points are a severely longer scraping time (on some of the 40 pages sections we're talking minutes worth)
Would it be possible to show more verbose feedback when it is scraping? Even on the curent version, it just displays a spinning 'loading' icon (Kodi skin dependent?), so when it takes a long time it is hard to tell if it has stuck or not. I wanted to give it a go myself but haven't been able to grok the codebase yet.
@Varstahl hey man your work is AWESOME! Haven't tried it yet (planning to later....), but wanted to say that if you needed a tester in India, I can! I'll give feedback as well! It's the least I can do!!!
(again, this is like MAGIC!!!111!!)
@Varstahl Why you removed the audiotrackid
parameter at getstreamurls method. Maybe this lets you only get English audio tracks. Did you checked the mpd manifest for additional audio tracks?
@ammgws in order to better debug things I removed a bit of unnecessary stuff from the logs, but you can enable the debug mode/verbose mode from the addon settings to show in the logs what it's actually fetching. I could probably have some feedback in the UI popping up, I just fear that it would be a terrible hassle.
@GNOME-IS-LIFE I don't think we have anyone testing from that region, so the more the merrier :)
@Sandmann79 ~there is no such parameter on primevideo, I manually checked every single query from different browsers, and even checked the MPD that was downloaded. The languages are all there, but the only parsed ones are the english, I can't tell you why. But sure, I'll re-enable it today for a quick test and see if it works.~ Well, I'll be damned, it works. I have no idea what changes, given that I manually inspected the downloaded MPD streams, but oh well, I've been working on the logs so much I could have mistaken the kodi ones with the browser ones, at this point I won't ever know. Thanks @Sandmann79 :D
you can enable the debug mode/verbose mode from the addon settings to show in the logs what it's actually fetching.
I'm not really troubled with it, it is more for the less technically adept members of the family. I was thinking of something unobtrusive like when Kodi updates the video library. If I get some time later I will try have a go.
Yeah, I'm still on the fence about the download times. Having runtimes and fanart are ok, but we're talking in some cases 80-100 additional HTTP requests per category. I think I might remove the option from movies and only use them for series where they're necessary. I'll have to look into that.
Edit: @ammgws, I just implemented what you requested in the primevideo.com branch, made perfect sense after all ;D
@Varstahl your awesome work, didn't work on my Kodi 18 nightly computer. I logged in fine, chose primevideo and out of that the region you said was for India (ROW_FE i think?) yet when I clicked any category, be it Movies or TV Shows or Kids, it spewed out an error in the logs.
my logs are attached
`08:12:39.912 T:140022730032896 WARNING: CPythonInvoker(1, /home/avi/.kodi/addons/plugin.video.amazon-test/default.py): the python script "/home/avi/.kodi/addons/plugin.video.amazon-test/default.py" has left several classes in memory that we couldn't clean up. The classes include: N9XBMCAddon9xbmcaddon5AddonE 08:12:42.074 T:140022730032896 ERROR: EXCEPTION Thrown (PythonToCppException) : -->Python callback/script returned the following error<--
@GNOME-IS-LIFE the source you're running is outdated, today alone I've pushed 12 updates, and I'm about to file a push request. You should either wait the addon update or download the default.py
, settings.xml
and resource.language.en_GB/strings.po
from my repo.
PR is merged at 3a31f7ab435cbc21e6dd99d52f41270264579e0e. Thanks @Varstahl for his good work.
Hi, Amazon Prime Video is now available also in Italy. Please, can you upgrade your addons to supports Amazon Prime Video for Italy?
Thanks!