search

iv-org / invidious

Invidious is an alternative front-end to YouTube
https://invidious.io
GNU Affero General Public License v3.0
16.39k stars 1.83k forks source link

Instance stops working when the cookie is added to the config.yml file #2320

Closed tio-trom closed 3 years ago

tio-trom commented 3 years ago

I think this new way of storing the captcha cookie that it is already implemented, stopped working and entire instances are not functioning anymore. Our ytb.trom.tf and these other instances that I suspect are dealing with the same issue: https://invidious.snopyta.org https://vid.puffyan.us https://invidious.namazso.eu - they output "video unavailable". If I am to delete the cookie line from the config file, it will work for a bit. The it stops once it generates that cookie line in the config. This stopped working some 2 days ago suddenly.

The logs do not show much. https://paste.trom.tf/otodipoles.pl

And stuff like

2021-08-10 01:25:00 UTC [error] get_video: mepxSzKfEhQ : Video unavailable. 2021-08-10 01:25:02 UTC [error] get_video: hW-vd2nQxqA : Video unavailable. 2021-08-10 01:25:20 UTC [error] get_video: VunlzEHdMGY : Video unavailable. 2021-08-10 01:25:32 UTC [error] get_video: eMudLfL76Sg : Video unavailable. 2021-08-10 01:25:37 UTC [error] UpdateDecryptFunctionJob : Nil assertion failed [development] Kemal is ready to lead at http://0.0.0.0:3000

A brief update: even if I disable anti-captcha in the config file, it still won't work. Same error. Only if I delete that cockie line will work again. This may help you debug it.

balasratran commented 3 years ago

Status Code: 403

We're sorry...
... but your computer or network may be sending automated queries. To protect our users, we can't process your request right now.

See Google Help for more information.
tio-trom commented 3 years ago

@balasratran are you saying that's the internal error for the above problem?

balasratran commented 3 years ago

@tio-trom Yes, google just made changes to prevent captcha decoding

tio-trom commented 3 years ago

ohh.....that's bad....

TheFrenchGhosty commented 3 years ago

@balasratran Google can't block captcha solver. Unless they stop giving captcha to flagged IP (highly unlikely considering instances that don't use anticaptcha still work perfectly) the captcha will be available and will be sent to anticaptcha.

If this issue is real, it's just that something broke in the way invidious handle anticaptcha.

balasratran commented 3 years ago

@TheFrenchGhosty Exactly this happened. Google returns a 403 response code and the captcha is not displayed, antidaptcha cannot be processed. I've been trying to figure out how to solve this problem for the past few days. screenshot_1628784449

ItsSt0ne commented 3 years ago

Here are some tests done with curl that may help to diagnose the issue, all done while the instance is blocked from viewing video pages (without captcha).

With no captcha cookie:

<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="https://www.google.com/sorry/index?continue=https://www.youtube.com/watch%3Fv%3DD_qFWoa_HR4&amp;q=EhAmBC3AAQECAAAAAAAAAA22GLqc2IgGIhApIGLS7hdCWcQOtAMeZBN0MgFy">here</A>.
</BODY></HTML>

With the captcha cookie in the configuration file after Invidious has determined it is blocked the first time, after "Video unavailable" errors occur:

<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"/><title>Sorry...</title><style> body { font-family: verdana, arial, sans-serif; background-color: #fff; color: #000; }</style></head><body><div><table><tr><td><b><font face=sans-serif size=10><font color=#4285f4>G</font><font color=#ea4335>o</font><font color=#fbbc05>o</font><font color=#4285f4>g</font><font color=#34a853>l</font><font color=#ea4335>e</font></font></b></td><td style="text-align: left; vertical-align: bottom; padding-bottom: 15px; width: 50%"><div style="border-bottom: 1px solid #dfdfdf;">Sorry...</div></td></tr></table></div><div style="margin-left: 4em;"><h1>We're sorry...</h1><p>... but your computer or network may be sending automated queries. To protect our users, we can't process your request right now.</p></div><div style="margin-left: 4em;">See <a href="https://support.google.com/websearch/answer/86640">Google Help</a> for more information.<br/><br/></div><div style="text-align: center; border-top: 1px solid #dfdfdf;"><a href="https://www.google.com">Google Home</a></div></body></html>

With a new captcha cookie obtained via using the instance's host as a web proxy (truncated, but in short this cookie worked):

<!DOCTYPE html><html style="font-size: 10px;font-family: Roboto, Arial, sans-serif;" lang="en" typography typography-spacing><head<!DOCTYPE html><html style="font-size: 10px;font-family: Roboto, Arial, sans-serif;" lang="en" typography typography-spacing><head
><meta http-equiv="X-UA-Compatible" content="IE=edge"/><script nonce="Zd+6LiAxl7+bnCGIeTCgQA">var ytcfg={d:function(){return window.yt&&yt.config_||ytcfg.data_||(ytcfg.data_={})},get:function(k,o){return k in ytcfg.d()?ytcfg.d()[k]:o},set:function(){var a=arguments;if(a.length>1)ytcfg.d()[a[0]]=a[1];else for(var k in a[0])ytcfg.d()[k]=a[0][k]}};
window.ytcfg.set('EMERGENCY_BASE_URL', '\/error_204?t\x3djserror\x26level\x3dERROR\x26client.name\x3d1\x26client.version\x3d2.20210812.00.00');</script><script nonce="Zd+6LiAxl7+bnCGIeTCgQA">(function(){window.yterr=window.yterr||true;window.unhandledErrorMessages={};window.unhandledErrorCount=0;
window.onerror=function(msg,url,line,columnNumber,error){var err;if(error)err=error;else{err=new Error;err.stack="";err.message=msg;err.fileName=url;err.lineNumber=line;if(!isNaN(columnNumber))err["columnNumber"]=columnNumber}var message=String(err.message);if(!err.message||message in window.unhandledErrorMessages||window.unhandledErrorCount>=5)return;window.unhandledErrorCount+=1;window.unhandledErrorMessages[message]=true;var img=new Image;window.emergencyTimeoutImg=img;img.onload=img.onerror=function(){delete window.emergencyTimeoutImg};

Looks like requests made with an expired captcha cookie may lead to a different page.

Also, these cookies seem to be expiring faster, though not sure if that's something that can be fixed.

tio-trom commented 3 years ago

Yah @SamantazFox is investigating this very closely. I gave her a captcha key so she can test with the official testing instance. They are working hard to find a solution very soon. Ofc any useful info should be posted here to help them out.

unixfox commented 3 years ago

Yah @SamantazFox is investigating this very closely. I gave her a captcha key so she can test with the official testing instance. They are working hard to find a solution very soon. Ofc any useful info should be posted here to help them out.

The solution is just #2220

unixfox commented 3 years ago

Although we haven't really fixed the original issue, with the merge of #2220 and #2255 you will never really get any CAPTCHA from Google anymore because we are now using an internal YouTube API that has no rate limiting.

I'm closing this issue as it is now kinda irrelevant.

tio-trom commented 3 years ago

Oh that's amazing! Thank you so much. So just to be clear, no need for using anti-captcha.com right?

unixfox commented 3 years ago

Oh that's amazing! Thank you so much. So just to be clear, no need for using anti-captcha.com right?

Until YouTube actively try to limit their internal API, you won't need anti-captcha anymore.