search

caronc / nzb-subliminal

Fetches subtitles for the videos it's provided. It can be easily integrated into NZBGet and SABnzbd too.
GNU General Public License v3.0
103 stars 14 forks source link

Adding more search providers from python project Sub.Zero #51

Open vledder opened 6 years ago

vledder commented 6 years ago

Hi,

I've using your subliminal work for a long time now very happily but always wondered if it could support more providers. Today I stumbled upon this python project (see below) and I was wondering if you could also incorporate (fork) the following branch to support more providers:

https://github.com/pannal/Sub-Zero.bundle/tree/master/Contents/Libraries/Shared/subliminal/providers

wondering upon your vision.

caronc commented 6 years ago

This is similar to your other question (#26).

I'll have a look at the link you shared and see how difficult it would be to merge some of the already done work by them.

The catch is that Sub-Zero just using the latest version of Subliminal (which was completely re-written) found here. Mine is still based off of the older version (the v0.7 branch). So these plugins won't necessarily just merge (although it's possible!).

The ideal solution going forward would be for me (specifically this script) to adapt to the new version of Subliminal and to stop trying to patch the older one. That would grant you all of the new providers but would break other functionality this script adds. I'll keep the ticket open without any immediate promises as this is a big task (but a good one none the less :smile:).

vledder commented 6 years ago

yeah, you can close #26 IMO. I guess #51 is more appropriate and more important, but I see now that your patches upon 0.7 need to be redone on the new master and is indeed of some work.

So basically there are two options:

wondering which strategy you gonna take. Anyway keep it up

caronc commented 6 years ago

I created a new branch here which is a first attempt on importing the following additional subtitle providers:

vledder commented 6 years ago

nice! I'll give it a testdrive! were the other (existing) providers also updated with latest patches or not? This so I know my testscope :)

other than that: I saw a patch for subcenter that fixes the broken state (somehow the provider got renamed) see: https://github.com/Diaoul/subliminal/pull/817

while I saw that I was intrigued by another patch for the most popular provider nowadays: https://github.com/pannal/Sub-Zero.bundle/blob/master/Contents/Libraries/Shared/subliminal_patch/providers/subscene.py

dunno if you could make it work though. Keep it up! Once I'll have some testresults I'll let you know.

caronc commented 6 years ago

I updated the new branch to support cinemast (http://www.cinemast.org/) as per your recommendation. Hopefully it works as i don't speak the language :wink:.

Keep me posted either way and i'll do my best to try to accommodate any issues you find. One thing I noticed is that cinemast requires a login and password and so does subcenter when/if it works.

other than that: I saw a patch for subcenter that fixes the broken state (somehow the provider got renamed)

I noticed that too. In the first commit I had already updated the URL to what it appeared to have changed to; but the site still doesn't appear to work properly (can't be sure though).

caronc commented 6 years ago

@vledder have you had a chance to try any of this out? I'd be curious to know what your feedback is.

vledder commented 6 years ago

I just came back from a break. I hope to get some testing time in upcoming days. I'll keep you posted

vledder commented 6 years ago

just had a quick first attempt; but getting an error:

update: I fixed the below (and follow-up import errors) by copying the dependencies from notify branch (decorator, markdown, requests_oauthlib, oauthlib)

will now start testing the providers one by one

Post-process-script nzbToMedia/Subliminal.py for Homeland_S07E11_720p_WEB_H264-DEFLAT failed (terminated with unknown status)
Subliminal: ImportError: No module named decorator
Subliminal: from decorator import decorator
Subliminal: File "/share/MD0_DATA/.qpkg/NZBGet/nzbget/scripts/nzbToMedia/Subliminal/apprise/plugins/NotifyPushjet/pushjet/utilities.py", line 7, in <module>
Subliminal: from .utilities import (
Subliminal: File "/share/MD0_DATA/.qpkg/NZBGet/nzbget/scripts/nzbToMedia/Subliminal/apprise/plugins/NotifyPushjet/pushjet/pushjet.py", line 8, in <module>
Subliminal: from .pushjet import Service, Device, Subscription, Message, Api
Subliminal: File "/share/MD0_DATA/.qpkg/NZBGet/nzbget/scripts/nzbToMedia/Subliminal/apprise/plugins/NotifyPushjet/pushjet/__init__.py", line 5, in <module>
Subliminal: from .pushjet import errors
Subliminal: File "/share/MD0_DATA/.qpkg/NZBGet/nzbget/scripts/nzbToMedia/Subliminal/apprise/plugins/NotifyPushjet/NotifyPushjet.py", line 19, in <module>
Subliminal: from . import NotifyPushjet
Subliminal: File "/share/MD0_DATA/.qpkg/NZBGet/nzbget/scripts/nzbToMedia/Subliminal/apprise/plugins/NotifyPushjet/__init__.py", line 17, in <module>
Subliminal: from .NotifyPushjet.NotifyPushjet import NotifyPushjet
Subliminal: File "/share/MD0_DATA/.qpkg/NZBGet/nzbget/scripts/nzbToMedia/Subliminal/apprise/plugins/__init__.py", line 37, in <module>
Subliminal: from .plugins.NotifyBase import NotifyBase
Subliminal: File "/share/MD0_DATA/.qpkg/NZBGet/nzbget/scripts/nzbToMedia/Subliminal/apprise/__init__.py", line 31, in <module>
Subliminal: from apprise import Apprise
Subliminal: File "/share/MD0_DATA/.qpkg/NZBGet/nzbget/scripts/nzbToMedia/Subliminal.py", line 421, in <module>
Subliminal: Traceback (most recent call last):
Executing post-process-script nzbToMedia/Subliminal.py for Homeland_S07E11_720p_WEB_H264-DEFLAT
vledder commented 6 years ago

Finished my test. Results we're surprsing... Since I'm not allowed to upload a rar/zip with bundled txt files I'm contacting you by mail.

caronc commented 6 years ago

@vledder I really appreciate all of your feedback! Thank you for testing all of this; i got your email loud and clear. In summary for on-lookers: I'm definitely not there yet! I think this branch shows breakage in other subtitle websites that used to work fine (and still do in the master branch). :flushed:

As per the error you're getting; that's on me (it's fixed in #54) but that's in the master branch (not the one you're testing from). I'll try to merge that asap. You can also just run the following to make decorator available to you globally too if you like:

# install decorator via pip
pip install decorator
caronc commented 6 years ago

I re-based this branch with all of the enhancements in the master one and made a few small changes. i'd be curious to see what kind of results you get.

hnernes commented 6 years ago

I have found this development fork of diaoul's subliminal by mamins1376:

https://github.com/mamins1376/subliminal/tree/develop/subliminal/providers

In this fork he have added subscene (wich is a really good provider). Is it possible for you to use some of this work in your re-based branch; or even better if you could use it in the master branch?

Will you pursue the updated base version any further (I assume that is the v2 branch)?

I have also found out (but I'm not sure of this) that opensubtitles.org have changed the return format from their API to dict(?). This have been fixed in kodi recently so I hope this is something that you can look into also, because as of now the opensubtitles.org is non functional for subliminal.

caronc commented 6 years ago

@hnernes Thanks for your input. Upon re-reading all of the comments above; @vledder actually asked for this provider too (and provided a link) above, but i some how missed it.

This ticket has become somewhat of a train-wreck because i'm developing it only speaking and understanding the English language. I'm relying on others to provide me input as to what is working and what isn't. I'll see what i can do using the source you provided in addition to the one already provided.

May i ask what language specifically you're looking to query? I'm asking you (or anyone else for that matter) seeing this post if they could just check out this branch and provide further feedback on the already added multi-language support.

hnernes commented 6 years ago

I'm mostly interrested in english subtitles because they are often the first to be of good quality, but my native language is norwegian, so I search for that also when they are available.

The providers that subliminal is supporting in its current state (master branch) is not very good, because I find that opensubtitles does not work. Subscene is a provider that is equal to opensubtitles as far as I know and would be a very good addition or replacement for opensubtitles.

I have actually tried that branch you are reffering to, but got an error with podnapisi (which is the only one currently returning results) and therefore that branch was not working for me. If you could check/fix opensubtitles and also add subscene to that branch, I will most certanly use and test that branch.

caronc commented 6 years ago

@hnernes I've re-based this whole branch to include everything in the master. I've just now completed the subscene portion i think... (it worked for me on the few tests I did anyway :wink:).

I'll be curious on your feedback!

hnernes commented 6 years ago

Thanks for your efforts on the subscene integration. I have now tested this, but I get this warning: "WARNING - Provider 'subscene' is not available, discarding it". I have tested my connection with subscene.com and I am getting ping returns from the site, so I should be able to reach it I think.

Opensubtitles are still returning 0 matches, even though I know there are plenty of matches on the movie I'm testing.

caronc commented 6 years ago

@hnernes maybe you could enable Debug mode or if you're using the CLI, add a -D switch? I'm able to fetch subs from subscene without a problem so that means there is something different between our two environments that is causing the problem.

As per Opensubtitles... i'm getting a zero return each time too. This is a really good find on your part! What's strange is the bug appears to be with their API. I tried to make the query a little smarter, but it just refuses to match on what i search for even though the site clearly has it.

The alternative to this would be to stop using the API and write a web-page scraper like the others are. This will be a lot more work though. Do you know of any other subtitle tools that are actually working against opensubtitles?

hnernes commented 6 years ago

@caronc I have attached my logfile with -D switch activated on a CLI run. I don't see much information as to why my environment doesn't reach subscene, but I see: "Request failed with status code 409" so there may be some room for improvement here. :)

Regarding opensubtitles: I know that Kodi have an opensubtitles addon that is working. According to a forum thread, the API returns from opensubtitles have changed from LIST to "DICT". ref: #16 in this thread: https://forum.kodi.tv/showthread.php?tid=326597&page=2 In there you will also find a service.py file, which supposedly have this issue fixed.

I have tried this addon for Kodi: https://github.com/amet/service.subtitles.opensubtitles and this works on a friends Kodi rig. It does however seem like this addon does not contain the fixed service.py, but it still works, so I am a bit confused about that (maybe Kodi isn't using API, but rather a web site scraper as you mentioned?).

I don't know how to check to either confirm or deny the change of API return, so I leave it up to you to figure this out, but I hope this will help somewhat in getting things on the right track again. :)

Logfile: logfile2.txt

caronc commented 6 years ago

A 409 HTTP Error Code ... I don't think I've ever seen that one issued before from a website. :confused: 

2018-06-26 09:02:52,566 - 4015 - DEBUG - ProviderNotAvailable error: 'Request failed with status code 409'

According to: https://httpstatuses.com/409

409 Conflict. The request could not be completed due to a conflict with the current state of the target resource. This code is used in situations where the user might be able to resolve the conflict and resubmit the request.

I honestly don't have any advice for you right now. When you google this error code, you get all sorts of wild and crazy possible answers (from banned IPs, to ISP interference, to dates in the web requests). :man_shrugging:

I notice you're using a bleeding edge version of Python here too (v2.7.15rc1). I will try to spin up a docker container this weekend and see if i can get my hands on this release candidate. What OS are you using right now? Maybe i can mirror your configuration?

hnernes commented 6 years ago

I use Ubuntu 18.04 Bionic Beaver. I usually update everything as soon as they appear, but I could downgrade Python if that could be the issue.

Yes, I also googled 409, but did not get any info I could use from that.

Have you looked some more into opensubtitles?

caronc commented 6 years ago

Have you looked some more into opensubtitles?

None yet; i keep getting zero results. I even get zero results if i log in. The information you shared still tells me I'm doing everything correctly (as far as i can tell unless i'm overlooking something).

I'll keep digging; but it's tough to tell at this point. They've documented it pretty well here too and doesn't shed any more light onto the problem. :disappointed:

I'll keep poking around though.

hnernes commented 6 years ago

I see in Sub-zero's changelog for version 2.5.3.2414: "providers: opensubtitles: add fallback for dict based query response in contrast to list/array based".

I don't know if you have looked into the DICT/LIST/ARRAY issue, but here's another prove for the change in response from opensubtitles.

There are also some other changes in the versions after this, addressing the opensubtitles issue.

https://github.com/pannal/Sub-Zero.bundle/blob/master/CHANGELOG.md

caronc commented 6 years ago

@hnernes and @vledder: aside from the opensubtitles not working (it's because of their API), have the other subtitles been working okay? I"m wondering if it's worth merging all of this code into the master branch or not?

I've reviewed the Sub-Zero code and I'm querying the servers the same way they are. So if it's not working for us; i can only presume it's not working for that module either. Can you confirm if this is the case?

Regardless... I'd like to open a separate ticket for this (opensubtitles) problem and tackle it independently from the addition of all of these other subtitle scrapers. Opensubtitles isn't working (for the same reason) in the master branch either (so it's not going to get any worse by pulling in this copy from this fork).

Thoughts?

hnernes commented 6 years ago

@caronc: The Subscene provider is not and have never worked for me unfortunately. I still get the message: "WARNING - Provider 'subscene' is not available, discarding it" with the DEBUG info: "ProviderNotAvailable error: 'Request failed with status code 409" like before.

I understand that this provider have worked for you, so there could be some problem with my ISP(?), but it seems unlikely since I am able to fetch subtitles manually from subscene.com. However: it appears like Sub-zero is NOT using subscene for movies, only for TV-shows, so I can't say whether or not subscene works in Sub-Zero for movies. (This is a bit strange considering this provider have an extremely large library of movie subtitles, but this is of topic).

Edit: Here's a link to why it's not used for movies: https://github.com/pannal/Sub-Zero.bundle/issues/551

I have a logfile from Sub-zero were podnapisi and opensubtitles is activated if you are interested. It seems like Sub-zero gets a lot of hits from opensubtitles, so there must be something you do differently.

com.plexapp.agents.subzero.log

Hope this helps. :)