search

snarfed / huffduff-video

📺 Extract the audio from videos on YouTube, Vimeo, and other sites and send it to Huffduffer.
https://huffduff-video.snarfed.org/
95 stars 6 forks source link

Has Vimeo blocked the IP addresses for huffduff-video? #33

Closed jrsinclair closed 6 years ago

jrsinclair commented 6 years ago

I've been having difficulty huffduffing Vimeo videos for a few months now. When I attempt to huffduff, I get error messages similar to this one:

Fetching https://vimeo.com/278439003 ...
ERROR: Unable to extract info section (caused by ExtractorError(u'Unable to download webpage: HTTP Error 404: Not Found (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Here are the supported sites. If this site isn't supported, it may also post its videos on YouTube. Try there!

I'm not sure exactly what is going on, but I can think of two plausible scenarios:

  1. Huffduff-video has triggered some automated spam blocker in Vimeo by sending lots of requests from the same IP range; or
  2. Vimeo has changed the URL format from which they serve their videos.

I suspect the former is more likely, but I don't actually know what's going on.

snarfed commented 6 years ago

ugh. sorry for the trouble, and thanks for reporting!

...weird. i was able to reproduce this. i confirmed that huffduff-video is on the latest youtube-dl release (2018.07.04; it updates automatically). i tried youtube-dl -v https://vimeo.com/278439003 locally on my laptop here at home, same version, and it worked ok. i poked at a few minor things, bounced apache on huffduff-video, tried the same command line there, and it unexpectedly worked...and now huffduff-video itself works too. it huffduffed your example video fine.

no idea what happened. :/

jrsinclair commented 6 years ago

Thanks so much for the amazingly quick reply! So far so good. 😸

jrsinclair commented 6 years ago

This one seems to have reared its ugly head again ☹️

Some sample Vimeo URLS:

Both showing that old 404 error. Really odd.

snarfed commented 6 years ago

whee. i see it too.

https://github.com/rg3/youtube-dl/issues/14332 (1y old) looks like the same problem. no solution there.

snarfed commented 6 years ago

@edsu reported the same thing in #34. reopening.

(fwiw huffduff-video always automatically updates itself to the latest youtube-dl version, currently 2018.7.21, so that's not the problem here.)

edsu commented 6 years ago

Does youtube-dl work ok on Vimeo URLs when you run it on the huffduff-video.snarfed.org machine? I guess that would test whether there is some IP blocking going on?

snarfed commented 6 years ago

@edsu last time no, then yes: https://github.com/snarfed/huffduff-video/issues/33#issuecomment-403329433

snarfed commented 6 years ago

ok, so, i tried https://vimeo.com/268653081 just now, and i think the same thing happened as before. i ran youtube-dl https://vimeo.com/268653081 locally, ie on a different IP than huffduff-video. it downloaded fine. i then tried it on huffduff-video, and it worked fine too.

odd. so maybe vimeo and/or huffduff-video get into some weird state with some videos, and if you then nudge them from a different IP, they work again?

edsu commented 6 years ago

is youtube-dl installed system wide on huffduff-video?

snarfed commented 6 years ago

@edsu yes. not sure how that would matter though.

i confirmed this same behavior just now on another video reported above, https://vimeo.com/279258468 . weird. i'm mentioning all this in https://github.com/rg3/youtube-dl/issues/14332 now.

snarfed commented 6 years ago

https://github.com/rg3/youtube-dl/issues/14332#issuecomment-407138925 :

we're seeing this right now too...but with an odd pattern. our service that uses youtube-dl, http://huffduff-video.snarfed.org/ , runs on an AWS EC2 instance. it's recently started hitting this error on some vimeo videos.

if i use youtube-dl to download the same vimeo video on my local machine, on a different ISP/network (usually residential or commercial), it works fine. then, after a few seconds, if i try http://huffduff-video.snarfed.org/ again, it doesn't hit the error and downloads the video fine.

so maybe vimeo is selectively blocking huffduff-video's IP address, maybe just on certain videos, but then when a different IP downloads that video, it removes the block? odd if so. not sure what to do about it.

snarfed commented 6 years ago

if they were blocking us, they've evidently stopped. huffduff-video successfully downloaded a vimeo video just now. tentatively closing.