search

iv-org / invidious

Invidious is an alternative front-end to YouTube
https://invidious.io
GNU Affero General Public License v3.0
16.47k stars 1.86k forks source link

Video unavailable #811

Closed simmstein closed 4 years ago

simmstein commented 5 years ago

Hello,

Versions

What happened

I click on a video's thumbnail and the message "Video unavailable." is shown.

Expected

Play the video clicked.

Information

The video is played using invidio.us (0.19.1-e03b4b7 master).

simmstein commented 5 years ago

From the server:

$ youtube-dl "https://www.youtube.com/watch?v=XXXXXXXXX"
[youtube] -OZ--RQcaY4: Downloading webpage
[youtube] -OZ--RQcaY4: Downloading video info webpage
WARNING: unable to download video info webpage: HTTP Error 429: Too Many Requests
WARNING: unable to download video info webpage: HTTP Error 429: Too Many Requests
WARNING: unable to download video info webpage: HTTP Error 429: Too Many Requests
WARNING: unable to download video info webpage: HTTP Error 429: Too Many Requests
WARNING: unable to download video info webpage: HTTP Error 429: Too Many Requests
Traceback (most recent call last):
  File "/usr/bin/youtube-dl", line 11, in <module>
    load_entry_point('youtube-dl==2019.1.17', 'console_scripts', 'youtube-dl')()
  File "/usr/lib/python3/dist-packages/youtube_dl/__init__.py", line 477, in main
    _real_main(argv)
  File "/usr/lib/python3/dist-packages/youtube_dl/__init__.py", line 467, in _real_main
    retcode = ydl.download(all_urls)
  File "/usr/lib/python3/dist-packages/youtube_dl/YoutubeDL.py", line 2002, in download
    url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "/usr/lib/python3/dist-packages/youtube_dl/YoutubeDL.py", line 793, in extract_info
    ie_result = ie.extract(url)
  File "/usr/lib/python3/dist-packages/youtube_dl/extractor/common.py", line 508, in extract
    ie_result = self._real_extract(url)
  File "/usr/lib/python3/dist-packages/youtube_dl/extractor/youtube.py", line 1672, in _real_extract
    token = video_info.get('token') or video_info.get('account_playback_token')
AttributeError: 'NoneType' object has no attribute 'get'

To be confirmed: this is not a issue from invidious.

atahanacar commented 5 years ago

12 hours ago, I had the same issue, but the issue is no longer there right now. Seems like Youtube changed some policies regarding connections.

Perflyst commented 5 years ago

Your IP got blocked by Google, other instances face the same issues.

An example to test if you are blocked:

root@invidious:~# curl 'https://www.youtube.com/watch?v=ukDht4jRcT0&gl=US&hl=en&disable_polymer=1&has_verified=1&bpctr=9999999999'
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="https://www.google.com/sorry/index?continue=https://www.youtube.com/watch%3Fv%3DukDht4jRcT0%26gl%3DUS%26hl%3Den%26disable_polymer%3D1%26has_verified%3D1%26bpctr%3D9999999999&amp;hl=en&amp;q=EgRf2BjmGJjv9uwFIhkA8aeDS2A2A_pZaQC7pukXkNR0Cy6xim-gMgFy">here</A>.
</BODY></HTML>
PureTryOut commented 5 years ago

I have the same error since like a week, but I get a full result if I try that test of @Perflyst.

simmstein commented 5 years ago

By using my server as a proxy, I had to fill recaptcha to play a video on youtube.com (private tab). By using curl, I had the full page and not a redirection.

OdinGitDat commented 5 years ago

I have the same for every video on v0.19.1-d36c536 vanilla docker container.

How do other instances get around this if it's a block on Google's end?

omarroth commented 5 years ago

You will also want to check the output for /das_captcha which will also indicate rate-limiting:

$ curl 'https://www.youtube.com/watch?v=ukDht4jRcT0&gl=US&hl=en&disable_polymer=1&has_verified=1&bpctr=9999999999' -s | grep '/das_captcha
<form action="/das_captcha" method="POST">

Testing with youtube-dl should return a 429 as well if this is the case.

As @Perflyst mentioned, this is due to rate-limiting/IP banning by Google. This has been an issue for other projects as well, see fent/node-ytdl-core#444, Tyrrrz/YoutubeExplode#292, and ytdl-org/youtube-dl#21729.

If IPv6 is available for your box, I would recommend testing with force_resolve: ipv4 or force_resolve: ipv6 in your config. Otherwise, you'll unfortunately have to wait for Google to unban you or get a new IP.

OdinGitDat commented 5 years ago

When I use a browser on my server to watch a video on youtube it asks for a captcha and then lets me watch videos for as long as the cookie stays in the browser.

Isn't there a way to relay this captcha to invidious webmasters and use the cookie for subsequent requests?

omarroth commented 5 years ago

It's possible. It's just not a practical workaround IMO since the cookie will only be valid for ~3 hours.

jaruiz59 commented 5 years ago

Yep getting the same message .Video unavailable.

asorel1942 commented 5 years ago

I don't quite understand what everyone in this thread means by hosting invidio locally. Is there a way to utilize the service without a browser? If so, will this be able to circumvent the "video is unavailable" messages everyone seems to be getting?

BurnyBoi commented 5 years ago

Does this mean invidio.us has been IP blocked by Google?

HumanG33k commented 5 years ago

Doing that they provide bad experience and people want to move back to YT. It’s a better option that blocking IP which provide badbuzz. They have done some similar things for ie when they downgrade qualities based on user agent in order to push FF users to Chrome.

OdinGitDat commented 5 years ago

You can always host it locally to avoid the IP banning.

@nchristensen Can you please stop spreading this as a solution? The OP specifically has this problem on his own instance and so do I. Yes I am the only user of my instance, I watch max 10 videos a day, and I've still been banned twice within 3 days. Both my bans lasted 12 hours and afaik they all do.

I don't like to be pessimistic, but I think this may be the end of invidious if relaying a captcha/cookie is impractical as omarroth said.

ghost commented 5 years ago

Why not redirect instance traffic to tor > towards youtube? The ip of the instance will be protected from being banned. Or use some open proxies and couple it all with proxychains and tor, it would be perfect !

simmstein commented 5 years ago

@coolcoderplus it could work but this is not a fix.

Gert-dev commented 5 years ago

@coolcoderplus I regularly use Tor and am often blocked by Google services due to "too much traffic". The list of endpoints is constantly changing, but likely the majority remains fairly constant. As also only a subset of Tor nodes function as actual endpoints, the pool of IP addresses is also said subset.

In a grossly simplifying sense, it's comparable to a dynamic list of proxies, of which the IP addresses end up flagged after a while.

sentriz commented 5 years ago

i am experiencing this on my home network too. youtube-dl doesn't work either. but surprisingly newpipe on my phone is working

2secslater commented 5 years ago

I have a somewhat public instance (other folk can access it) running on my home network, and has been for rougly three days now. I have had the same IP address, using the instance most of the time and I haven't been sniped by Google (yet). My instance is running commit 7b77f20

atahanacar commented 5 years ago

Invidious and youtube-dl are not working on my private instance running on a VPS. However, when I connect to the VPN hosted on the same VPS, I can access Youtube and youtube-dl works fine.

m4teh commented 5 years ago

I've also been hit by this (self hosting Invidious via outbound VPN provider). They must have done a big VPN sweep recently. I get a CAPTCHA when I browse YouTube's website too.

Is work being put into adding an ability to perform the CAPTCHA from the Invidious interface? That's the easiest solution I can think of.

inhji commented 5 years ago

My solution workaround was to snapshot the whole instance and restore it to a different VPS with a new ip address.

HumanG33k commented 5 years ago

it s hard workaround not really easly setup for people with less money/skills.

I guess there is no way to contact google for help us … they probably say use the api and we loose some privacy improvement provide by invidious.

atahanacar commented 5 years ago

@HumanG33k This is kind of unrelated but I am curious. If I am using an off-site server as a proxy, does it matter if I am using the API or an extractor?

HumanG33k commented 5 years ago

@atahanacar hope it can help :

if i m not wrong yes you can use an off-site server as a proxy. For me there is differences between an extractor and an api call if it s not open(need an auth). You also need to know if you are alone or not.

if i understand well the server make request to google insteed of you.

First case (standard usage) : you (auth or not) --> google login or not google know who you are.

Case 2 : you && server --> google google know 1 or more people comes from here (on same ip) its quite easy to know what you see if you are alone or other people don’t use the server but if you don’t use some tools (addon/ads blocker) google can know who you are.

Case 3 : you --> server --> google It s a little more difficult for google to know who you are but if he knows you use the server and nobody else then he knows.

Case 4 : you --> server --> random --> google this can be the better but also the hard to maintain case. to be less track (the main goal of google) you should never be the you google knows. the random should also be a several random, for me the most usefull random should be multiple random, between 1 to N by channel or topic. all random should also be used by several people (vpn, tor, for ie)

For the api vs scrapping the main issue with the api is you need to be auth in most of the case and 1 auth = 1 account. And google can know how many people exist in the worst case. (how many people look videos at the same time for ie) but api are created in order to provide data not fancy (useless) stuff like ui.

Crawler in other way use the (browser) human way to grab content but because fancy things are provide the content of a page can change and it can make tools which download hard to maintain.

For me the good way should be

you --> the One app --> server : with p2p and federation, and can mirror(grab and store) content (peertube or mediagobelin) --> random --> multiple providers (vimeo, YT, dailymotion, p2p, etc)

Pentaphon commented 5 years ago

Many videos are showing this for me now.

BurnyBoi commented 5 years ago

Started my own instance a week ago that only I use, and now this is already happening to me. Not an IP ban since I can use mpv to watch YT videos on the same IP just fine. This is crazy. Google seems to have some way of knowing that a video is being requested by an Invidious server. Can we hide that?

HumanG33k commented 5 years ago

i think about that last days.

i don’t know how @omarroth send request to YT but maybe we have to improve the fake network request. And fake at if we are Firefox or other but maybe not on a chromium/chrome base. I’m pretty sure they have some proprietary request come from chrome (with token, etc). they probably also have some other UI things to be prevent scraping. I also see a survey on a not loggin youtube page but i have to leave and i don’t do it. I think they maybe try again to kill FF. That can interesting to set up an MITM to check network packets from/to youtube.

Perflyst commented 5 years ago

What would that change? YouTube also blocks you if there are too many requests from a normal IP with a normal browser (see VPN or Tor network)

atahanacar commented 5 years ago

What would that change? YouTube also blocks you if there are too many requests from a normal IP with a normal browser (see VPN or Tor network)

I have a private instance that only I and a few friends use. It is blocked, but not by IP, as I can use my OpenVPN to watch YouTube videos successfully. The only reason I can't use Invidious is because of this silly block. I didn't even make that many requests, 10 videos a day at max.

HumanG33k commented 5 years ago

I can not use my invidious instances but youtubes with the same exit ip address sometimes. i will make test to see that.

I also suspect they have an ai for filter incoming request. If it can learn we are legitimate trafic it can be good.

For how google manage all things https://www.bloomberg.com/news/articles/2019-10-23/google-accused-of-creating-spy-tool-to-squelch-worker-dissent

elypter commented 5 years ago

maybe invidio.us should consider mirroring more metadata to minimize the request count. the distributed effort to rescue the annotations was quite successful.

jee-r commented 5 years ago

we'd need to put P2P in there :rocket:

elypter commented 5 years ago

there already are distributed videoplatforms like bitchute. no need to reinvent the wheel. its probably hard enough to integrate them into the project and the biggest problem is that as soon as you no longer just proxy content but host it you have to manage it and are legally responsible. no matter if its invidio.us who is responsible or the individual p2p users doesnt matter.

if it's just the metadata then thats a lot easier. and just the metadata can work without p2p as well

defterade commented 5 years ago

Started getting this on all videos today. I don't appear to be IP banned (youtube-dl, mpv, curl showed no issues).

Running latest flourgaz/invidious.

BurnyBoi commented 5 years ago

As an update to mine above, seems @m4teh is on the right track here. I noticed if I go to youtube.com directly on my server and go to watch a video, I get a recaptcha before the video will play. Obviously the Invidious instance is currently unable to solve this recaptcha. if Invidious could embed the recaptcha on the page, allow the user to solve it, and pass the result back to YouTube, this seems like it would do the trick. Also, it seems like once you store some cookie saying you solved it, it doesn't ask again, so maybe Invidious could do that too?

NettoHikari commented 5 years ago

I'm also hit by a ban since today. I'm hosting invidious since 2 days for me, my wife and 2 or 3 friends. It is publicly available, though. youtube-dl also doesn't work on the VPS the Invidious instance is running on. Very sad.

Edit: I thought about setting up TOR for Invidious only in conjunction with a cronjob script that checks if requests to YouTube fail. If the request fails, reconnect. Another way (for me) would be to host it locally in my own network, as my provider gives me a dynamic IP anyway. That, paired with a script to periodically reconnect my router would do... But really, both solutions are dumb as hell. I don't even know if my plan with TOR would work anyway. Just an idea.

cy8aer commented 5 years ago

The captcha problem looks like this really old one in searx: https://github.com/asciimoo/searx/issues/729

cy8aer commented 5 years ago

probably useful? https://github.com/unixfox/antibot-proxy

On the searx side the google module is blocked because bots try to use searx and do some brute forcing that let google block the searx side. This proxy filters out these bots.

ghost commented 5 years ago

Magically works again for me today, after a few weeks. Temp ban removed, maybe?

defterade commented 5 years ago

Magically works again for me today, after a few weeks. Temp ban removed, maybe?

For me too.

m4teh commented 5 years ago

Did you guys update Invidious? I saw some recent commits for captcha/user agent which may have a positive effect?

I’ll have to switch my VPN tunnel back to check if mine is still blocked or not.

On 31 Oct 2019, at 5:38 am, defterade notifications@github.com wrote:

Magically works again for me today, after a few weeks. Temp ban removed, maybe?

For me too.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

ghost commented 5 years ago

No update.

defterade commented 5 years ago

I didn't update either.

2secslater commented 5 years ago

No. That's an issue with the Chrome browser on client-ends.

cy8aer commented 5 years ago

Sorry I was too slow with deleting the post but because @2secslater answered I put it in again (but off topic)

Related?

https://www.bleepingcomputer.com/news/google/chrome-78-disables-code-integrity-check-to-mitigate-aw-snap-crashes/

simmstein commented 5 years ago

Without updating invidious, I can watch videos from my instance…For how much longer?

BurnyBoi commented 5 years ago

force_resolve: ipv4 Seemed to fix it for me...for now. So I guess they banned/limited my ipv6 address, but not ipv4.

Perflyst commented 5 years ago

A lot of instances started working again (v4 and v6). Either Google was testing some new "bot detection" or they had a bug. Nobody knows. Please do not spam this issue that yours is working again, thanks. For further discussion you can also join #invidious on freenode or #invidious:matrix.org

BurnyBoi commented 5 years ago

A single message isn't spam lol. Close it then

EDIT: If you're going to down-vote, at least provide proof that a single, concise message somehow qualifies as spam, or prove why a github issue that already has the solution in it shouldn't be closed :)