Subscene provider stopped working

[Ash-Raimon]

Describe the bug Subscene provider was working with the development branch v0.7.5. However, 2days ago it stopped working as it is no longer fetching subtitles, and will not show up with manual search.

To Reproduce Steps to reproduce the behavior:

1. When having subscene as one of the providers, choose any movie/TV show
2. Click on manual search
3. All included providers will show except Subscene

Expected behavior Subscene should've been included with automatic and manual search. I have also used Anti-Captcha with Subscene, it was working previously.

Software (please complete the following information):

• Bazarr: v 0.7.5
• Radarr v 0.2.0.1344
• Sonarr v 2.0.0.5322
• OS: Windows 10 64bit

[morpheus65535]

@Pannal you want to take a look please?

[morpheus65535]

Can you test dev branch which include a fix for this?

[Ash-Raimon]

Will test it. Anti-Captcha changed their ways, as of right now they need proof of document/id to be sent to them for their services to work. but that's a problem for another day, just need to change to death by captcha provider to work with subscene.

will reply back if it works

[Ash-Raimon]

Just to update, I've tried again by using Death By Captcha, and searched again for subscene provider. I didn't receive any subtitle from subscene with manual search. Here's a copy of subscene log. (v0.7.5.1)

10/06/2019 22:42:54|DEBUG |urllib3.connectionpool |https://subscene.com:443 "POST /subtitles/searchbytitle HTTP/1.1" 400 None|

10/06/2019 22:42:54|ERROR |subliminal_patch.core |Unexpected error in provider 'subscene': Traceback (most recent call last): File "c:\bazarr\bazarr../libs/subliminal_patch\core.py", line 162, in list_subtitles_provider results = self[provider].list_subtitles(video, provider_languages) File "c:\bazarr\bazarr../libs/subliminal_patch\providers\subscene.py", line 151, in list_subtitles return [s for s in self.query(video) if s.language in languages] File "c:\bazarr\bazarr../libs/subliminal_patch\providers\subscene.py", line 213, in query film = search(term, session=self.session, release=False, throttle=self.search_throttle) File "c:\bazarr\bazarr../libs/subscene_api\subscene.py", line 280, in search session=session) File "c:\bazarr\bazarr../libs/subscene_api\subscene.py", line 74, in soup_for ret.raise_for_status() File "c:\bazarr\bazarr../libs/requests\models.py", line 940, in raise_for_status raise HTTPError(http_error_msg, response=self)HTTPError: 400 Client Error: Bad Request for url: https://subscene.com/subtitles/searchbytitle|Traceback (most recent call last): File "c:\bazarr\bazarr../libs/subliminal_patch\core.py", line 162, in list_subtitles_provider results = self[provider].list_subtitles(video, provider_languages) File "c:\bazarr\bazarr../libs/subliminal_patch\providers\subscene.py", line 151, in list_subtitles return [s for s in self.query(video) if s.language in languages] File "c:\bazarr\bazarr../libs/subliminal_patch\providers\subscene.py", line 213, in query film = search(term, session=self.session, release=False, throttle=self.search_throttle) File "c:\bazarr\bazarr../libs/subscene_api\subscene.py", line 280, in search session=session) File "c:\bazarr\bazarr../libs/subscene_api\subscene.py", line 74, in soup_for ret.raise_for_status() File "c:\bazarr\bazarr../libs/requests\models.py", line 940, in raise_for_status raise HTTPError(http_error_msg, response=self)HTTPError: 400 Client Error: Bad Request for url: https://subscene.com/subtitles/searchbytitle|

[morpheus65535]

@pannal I've just tried on my own instance of Bazarr and I can reproduce it:

Unexpected error in provider 'subscene': 
Traceback (most recent call last):
  File "/app/bazarr/bazarr/../libs/subliminal_patch/core.py", line 162, in list_subtitles_provider
    results = self[provider].list_subtitles(video, provider_languages)
  File "/app/bazarr/bazarr/../libs/subliminal_patch/providers/subscene.py", line 151, in list_subtitles
    return [s for s in self.query(video) if s.language in languages]
  File "/app/bazarr/bazarr/../libs/subliminal_patch/providers/subscene.py", line 213, in query
    film = search(term, session=self.session, release=False, throttle=self.search_throttle)
  File "/app/bazarr/bazarr/../libs/subscene_api/subscene.py", line 280, in search
    session=session)
  File "/app/bazarr/bazarr/../libs/subscene_api/subscene.py", line 74, in soup_for
    ret.raise_for_status()
  File "/app/bazarr/bazarr/../libs/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)HTTPError: 400 Client Error: Bad Request for url: https://subscene.com/subtitles/searchbytitle

Of course this URL doesn't seems to be valid. Can you take a look at it? Thanks.

[pannal]

How is this url not valid? That's their new POST endpoint. Unfortunately they require recaptcha everywhere now.

I'll take a look once I get back.

[morpheus65535]

That’s what I get when I try to load the page:

[pannal]

Interesting. They might change their endpoints regularly now. I'll take a look soon

[morpheus65535]

Thanks!

[Ash-Raimon]

Thank you

[morpheus65535]

@pannal Did you had time to take a look at this one?

[pannal]

A little. This is getting serious.

[morpheus65535]

Can you try the dev branch? Don't forget to define a username and password for Subscene in settings and provide an anti-captcha provider.

[pannal]

The dev branch doesn't have the latest fixes yet. I'm still testing it locally. I'm talking out of my a** :)

[Ash-Raimon]

lol. Thanks for the efforts =) will give a an update to the latest dev branch and test it out.

[Ash-Raimon]

Subscene now works with the new update! Although its taking around 40-50sec for it to load the subtitles (when manually searching). when checking the logs, it is not using captchas as of testing it at this moment (also checked with my captcha provider), and in logs, subscene says its "Re-using old subscene cookies:" dunno if its relevant or not.

Thanks again!

[Ash-Raimon]

bazarr (1).log

this is the debug log.

[morpheus65535]

@pannal Check his log please. Why Subscene provider return subtitles for a lot more language that what was requested? It's clearly not optimal. We are hitting their website a lot and parsing way to much HTML...

[pannal]

I don't have the faintest clue why he searches without a language filter tbh.

[Ash-Raimon]

@pannal Just came back from work, and tried a different approach, it may sound stupid but I actually went into the site itself subscene.com and used their filter, to only filter 2 languages I needed. That's the only thing different from yesterday's test and today. And it worked.

bazarr (5).log

from the logs, it re-used my old cookies, and fetched the filtered languages. I know bazarr uses API to fetch, and it may have no relation to the actual website's filters.. I have no explanation lol

I used the same exact media file I used before, the same file was tested now and yesterday. The search took approximately 20-30secs, more than half the time than before (as there was a lot of different subtitles on the same language filtered for the same media, which is expected I guess)

I'll try more tests on different media's, And will see the logs. Maybe the last test had a cached bug or something dunno.

[Ash-Raimon]

Ok this time I went back to subscene.com and unfiltered from the site itself, tried the test again and now the log is fetching every language there is from the same media file I used.

here's the log:

bazarr (6).log

it took around 45secs (give or take) for it to fetch manual search via bazarr.

So i'm assuming with subscene, a temp fix would be that the user has to go on the site and filter the language they need.

Hope this helps out

EDIT: I also just tried going to subscene.com site, logging out and filtering what I need (thought it would be stored in cookies), and used bazarr to manual search subtitle for the same media file. It did not filter. and showed all languages regardless of the filter I had on the site while logged out. bazarr (7).log

So, a user has to be logged in on the site itself, and using the filter language on the site.

EDIT2: subscene provider also did not use any captcha, its working perfectly, and restarted bazarr several times, my captcha count is still 0 from subscene

[Ash-Raimon]

This has now been fixed thanks to pannal!