[Scraper] Import all IMDb keywords

[levander051]

Hi,

Any way to remove limit to imdb keywords import? Or at least make it optional?

[bugwelle]

What do you mean by "limit"?

Currently all keywords are imported that are shown on the movie detail page, e.g.
https://www.imdb.com/title/tt2024544/?ref_=fn_al_tt_1
5 keywords are shown and imported. You can list all 106 in this case, if you follow a link "See All":
https://www.imdb.com/title/tt2024544/keywords?ref_=tt_stry_kw
Do you mean those other 101 keywords?

[levander051]

Yes, that's what I mean by limit, I mean all the keywords! Those on the movie detail page are not always relevant to me.

Also, only these very popular movies usually have 100+ keywords.

So is there any way to enable import of all the keywords please?

[bugwelle]

No, currently there is no way to import all keywords as MediaElch just scrapes the movie page when using IMDb. I'll mark this issue as a feature request.

[bugwelle]

@levander051 I'd like to ask for your opinion on this. Should we import all keywords or only those which at least one person finds useful?
Because if I look at e.g. https://www.imdb.com/title/tt0371746/keywords keywords like jarvis character don't seem to be useful at all.

[levander051]

I would definitely go with all keywords. That is only way that makes sense.

Personally I would just avoid to import ANY keywords for kinds of movies like blockbusters. If I am Marvel Comics movie fan for example (I'm not) and there are cca 50 movies in that collection I WOULD BE HAPPY to personally enter and maintain keywords for those. I am more interested in those other cca 15000 movies I have in my library and I guess that some of them have at most 20 and some 200 while some doesn't have keywords at all. But there are many keywords that overlap. Believe it or not "jarvis character" keyword is contained in 5 titles on imdb.

P.S.

I am aware that "useless keywords" is relative. Something that I might use for keyword someone else will not. But again, I think that eliminating keywords (that in vast majority overlaps) is far more convenient than manually adding keywords to unwatched "unknown" movies in your library that might become more interesting when you can connect that movie to some "unknown" movie that you really like. Anyway, It could be also good idea to make a "blacklist" of keywords that you always want to avoid , for example: "frontal nudity" or "jarvis character".

[bugwelle]

I created a PR (#472) which allows the user to load all tags/keywords. For this I've added a new checkbox "Load all tags". See screenshot below:

Here is a movie for which I loaded all keywords using IMDb:

Is this okay for you?

[levander051]

Wow. Am I dreaming? Yes! Thank you!

[bugwelle]

I've merged my pull request. A new nightly version will be available in about 30min 😃

[levander051]

Wait!? What nightly version? From where and how can I install that on my Mac?

[levander051]

This probably:

https://bintray.com/komet/MediaElch/MediaElch-macOS/_latestVersion

right?

[bugwelle]

Exactly :) My changes are now available in the latest version. If you have any problems with it, please let us know.

[levander051]

Hey, it's working great!

A question. Would it be possible to add this feature to tv shows? I see tv shows tags are supported but there is no scraper for tags. I think it's logical to implement it as well.

Thanks.

[bugwelle]

I've added TV show tag scraping using IMDb in #473 You'll need to set IMDb for tag scraping in Settings -> Scraper -> TV Scraper -> Tags.

Currently only the top 5 tags are scraped. Loading all tags will come soon :)

PS: I've also fixed top250 for TV shows when the scraper for ratings is set to IMDb.

[levander051]

Great! Thanks.

P.S. Any news on #471?

[bugwelle]

I've responded to you there :) I didn't look into it, yet.

[levander051]

Hi, I've been using tags scraping these last 7 days and everything works fine except: if any of loaded films (for example 100 titles) scraped all the tags (it doesn't have to be 400, it' happening with 20) ui starts to responds really, really slow. Of course the more titles are loaded and the more tags are scraped ui is responding slower and slower. And it seems odd to me because, it's not so much more data on the stake. I have a workaround not to get stuck with slow ui: I scrape all tags when I finished editing everything else. Not convenient but for now it is useful for me. Btw, TMM loads all those movies with all those scraped tags without slowing down a bit.

[bugwelle]

Thank you for coming back to us :smiley: I think it is because of how MediaElch handles tags. All tags for every movie are shown in the "Extended" tab. Not only those that were actually downloaded. This means that when you select a movie, possibly thousands of tags have to be rendered.

I've opened #488

[levander051]

Hi :) Thanks for your response..

Hm, yes, that's what I thought. Well I guess changing the way ME is handling the tags is something to consider in long run but in the meantime something maybe should be done... I'm not programmer but I can guess that maybe:

1. it is possible to change that these tags should start to load WHEN extended tab is selected and not by default when selecting each movie.
2. this system could be taken advantage of in a way: since all tags that appear in loaded db are shown in extended tab, there should be option to kill (delete) certain tag in all movies. Hm?

[bugwelle]

@levander051 I've added a dirty workaround for this in #584. See my comment in https://github.com/Komet/MediaElch/issues/488#issuecomment-420372548

[levander051]

Hey, thanks! I will try how it works