search

Tzahi12345 / YoutubeDL-Material

Self-hosted YouTube downloader built on Material Design
MIT License
2.59k stars 265 forks source link

[BUG] Proxy Usage for Downloading is broken #177

Open tophee opened 4 years ago

tophee commented 4 years ago

Blocks: #230

My 429 errors seem to be gone now, but all my downloads keep failing.

Starting with all custim arguments:

2020-07-23T20:57:50.716Z VERBOSE: youtube-dl args being used: -o,video/%(title)s.mp4,--write-info-json,--print-json,-f best[ext=mp4],--cookies,appdata/cookies.txt,--add-metadata,--no-mtime,--write-description,--embed-thumbnail
2020-07-23T20:57:51.739Z DEBUG: URL info retrieval delay: 2.231 seconds.
2020-07-23T20:57:51.739Z ERROR: Error during parsing:Error: Command failed with exit code 1: /app/node_modules/youtube-dl/bin/youtube-dl --dump-json http://www.youtube.com/watch?v=kqqGr9bYm3o
2020-07-23T20:57:51.739Z ERROR: failed to parse for urls starting with https://www.youtube.com/watch?v=kqqGr9bYm3o
2020-07-23T20:57:55.273Z ERROR: Command failed with exit code 1: /app/node_modules/youtube-dl/bin/youtube-dl --dump-json -f best -o video/%(title)s.mp4 --write-info-json --print-json -f best[ext=mp4] --cookies appdata/cookies.txt --add-metadata --no-mtime --write-description --embed-thumbnail http://www.youtube.com/watch?v=kqqGr9bYm3o

Then taking out the custom args:

2020-07-23T20:59:29.895Z VERBOSE: youtube-dl args being used: -o,video/%(title)s.mp4,--write-info-json,--print-json,-f best[ext=mp4],--cookies,appdata/cookies.txt
2020-07-23T20:59:32.086Z ERROR: Command failed with exit code 1: /app/node_modules/youtube-dl/bin/youtube-dl --dump-json -f best -o video/%(title)s.mp4 --write-info-json --print-json -f best[ext=mp4] --cookies appdata/cookies.txt http://www.youtube.com/watch?v=kqqGr9bYm3o

And then even the cookies:

2020-07-23T21:02:08.852Z VERBOSE: youtube-dl args being used: -o,video/%(title)s.mp4,--write-info-json,--print-json,-f best[ext=mp4]
2020-07-23T21:02:10.994Z ERROR: Command failed with exit code 1: /app/node_modules/youtube-dl/bin/youtube-dl --dump-json -f best -o video/%(title)s.mp4 --write-info-json --print-json -f best[ext=mp4] http://www.youtube.com/watch?v=kqqGr9bYm3o

I can download the same video with youtube-dl on my windows PC in the same network without problems...

The log settings ate on debug, so I have no idea why there's not more showing up in the log.

Edit: And yes, I did restart the container.

Tzahi12345 commented 4 years ago

Can you try going into the Downloader tab in the settings and enabling "Safe download override"? I think the regular method might be bugged on some systems for some videos (worked fine on Windows, but reproduced the error on Ubuntu 18.04)

tophee commented 4 years ago

I now activated "Safe download override" but now I'm back to the 429 error:

2020-07-24T17:38:51.935Z VERBOSE: Download is running with the safe download override.
2020-07-24T17:38:51.937Z VERBOSE: youtube-dl args being used: -o,video/%(title)s.mp4,--write-info-json,--print-json,-f best[ext=mp4],--add-metadata,--no-mtime,--write-description,--embed-thumbnail
2020-07-24T17:38:57.064Z DEBUG: URL info retrieval delay: 7.073 seconds.
2020-07-24T17:38:57.065Z ERROR: Error during parsing:Error: Command failed with exit code 1: /app/node_modules/youtube-dl/bin/youtube-dl --dump-json http://www.youtube.com/watch?v=RG1iYRQtjKI
2020-07-24T17:38:57.065Z ERROR: failed to parse for urls starting with https://www.youtube.com/watch?v=RG1iYRQtjKI
2020-07-24T17:38:59.067Z DEBUG: Video download delay: 7.131 seconds.
2020-07-24T17:38:59.067Z ERROR: ERROR: Unable to download webpage: HTTP Error 429: Too Many Requests (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

In the mean time I have set up a VPN proxy but it doesn't work:

2020-07-24T17:47:24.458Z VERBOSE: Download is running with the safe download override.
2020-07-24T17:47:24.459Z VERBOSE: youtube-dl args being used: -o,video/%(title)s.mp4,--write-info-json,--print-json,-f best[ext=mp4],--add-metadata,--no-mtime,--write-description,--embed-thumbnail,--proxy,http://192.168.1.4:8888
2020-07-24T17:47:27.741Z DEBUG: Video download delay: 3.283 seconds.
2020-07-24T17:47:27.741Z ERROR: ERROR:

It took me a while to figure out that I need to submit the option and its arguments as two separate options: --proxy,,http://192.168.1.4:8888 but I still seem to be doing something wrong...

I have no idea where the 429s come from again. I haven't downloaded anything for about 24 hours...

Tzahi12345 commented 4 years ago

Is the proxy set up at a different public IP address? If it's the same IP it'll still be blocked. If you have access to a VPN service with SOCKS5 functionality, that would be ideal.

Anyways, I still need to solve the underlying "exit code 1" problem which seems unrelated to the 429 errors. I'll do some investigating on both ends tomorrow!

tophee commented 4 years ago

Yes, the whole point of having the proxy is that the proxy is going through a VPN. So youtube is not seeing the same IP.

Tzahi12345 commented 4 years ago

Right, just making sure. You're still getting 429's? At this point it's pretty weird it hasn't gone away..

tophee commented 4 years ago

Today I deleted the container and rebuild it with the latest image. 

2020-08-05T19:37:37.646Z INFO: YoutubeDL-Material v4.0 started on PORT 17442
2020-08-05T19:37:38.081Z INFO: Found new update for youtube-dl. Updating binary...
2020-08-05T19:37:39.919Z INFO: Binary successfully updated: 2020.06.16.1 -> 2020.07.28

This is what I got when trying to download a video (without proxy):

2020-08-05T20:20:46.418Z VERBOSE: Download is running with the safe download override.
2020-08-05T20:20:46.419Z VERBOSE: youtube-dl args being used: -o,video/%(title)s.mp4,--write-info-json,--print-json,-f best[ext=mp4],--add-metadata,--no-mtime,--write-description,--embed-thumbnail
2020-08-05T20:20:46.773Z DEBUG: URL info retrieval delay: 3.255 seconds.
2020-08-05T20:20:50.822Z DEBUG: Video download delay: 4.403 seconds.
2020-08-05T20:20:50.822Z ERROR: ERROR:
Tzahi12345 commented 4 years ago

Darn youtube-dl and its useless error messages... I'm kinda lost on this one. Can you try downloading this reddit video and see if it gives you any errors? https://www.reddit.com/r/battlefield_4/comments/i4ajcd/explosive_defibrillator/

Tzahi12345 commented 4 years ago

This may be cookies messing with your downloads (old cookies caused weird errors for me), can you see if renewing them or disabling them helps?

adan89lion commented 4 years ago

I now activated "Safe download override" but now I'm back to the 429 error:

2020-07-24T17:38:51.935Z VERBOSE: Download is running with the safe download override.
2020-07-24T17:38:51.937Z VERBOSE: youtube-dl args being used: -o,video/%(title)s.mp4,--write-info-json,--print-json,-f best[ext=mp4],--add-metadata,--no-mtime,--write-description,--embed-thumbnail
2020-07-24T17:38:57.064Z DEBUG: URL info retrieval delay: 7.073 seconds.
2020-07-24T17:38:57.065Z ERROR: Error during parsing:Error: Command failed with exit code 1: /app/node_modules/youtube-dl/bin/youtube-dl --dump-json http://www.youtube.com/watch?v=RG1iYRQtjKI
2020-07-24T17:38:57.065Z ERROR: failed to parse for urls starting with https://www.youtube.com/watch?v=RG1iYRQtjKI
2020-07-24T17:38:59.067Z DEBUG: Video download delay: 7.131 seconds.
2020-07-24T17:38:59.067Z ERROR: ERROR: Unable to download webpage: HTTP Error 429: Too Many Requests (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

In the mean time I have set up a VPN proxy but it doesn't work:

2020-07-24T17:47:24.458Z VERBOSE: Download is running with the safe download override.
2020-07-24T17:47:24.459Z VERBOSE: youtube-dl args being used: -o,video/%(title)s.mp4,--write-info-json,--print-json,-f best[ext=mp4],--add-metadata,--no-mtime,--write-description,--embed-thumbnail,--proxy,http://192.168.1.4:8888
2020-07-24T17:47:27.741Z DEBUG: Video download delay: 3.283 seconds.
2020-07-24T17:47:27.741Z ERROR: ERROR:

It took me a while to figure out that I need to submit the option and its arguments as two separate options: --proxy,,http://192.168.1.4:8888 but I still seem to be doing something wrong...

I have no idea where the 429s come from again. I haven't downloaded anything for about 24 hours...

In regard to the proxy issue, I can confirm that --proxy flag didn't work (at least on my instance). I've set a valid proxy (where I usually use with my youtube-dl scripts), but YoutubeDL-Material is still using host connection to download videos, not proxy's (as seen in DNS requests)

Tzahi12345 commented 4 years ago

@adan89lion The proxy arg fails even with the safe download override enabled?

adan89lion commented 4 years ago

@Tzahi12345 Proxy arg works after enabling Safe Download Override. Thanks for the info.

Tzahi12345 commented 4 years ago

@adan89lion Cool! We already have a way to force safe download mode when it recognizes args that won't work the normal way, so I went ahead and added --proxy to that, see commit here.

vista-narvas commented 3 years ago

im getting the same error
using docker tzahi12345/youtubedl-material:nightly
latest was giving me trouble with the subscriptions not showing up witch worked in nightly
but now i get this error using both latest or nightly 

2020-10-29T15:23:50.069Z VERBOSE: youtube-dl args being used: -o,users/admin/video/%(title)s.mp4,--write-info-json,--print-json,-f,bestvideo+bestaudio,--merge-output-format,mp4,--write-thumbnail, --proxy socks5://tor.proxy.local:9150
2020-10-29T15:23:54.898Z ERROR: Error while retrieving info on video with URL https://www.youtube.com/watch?v=92g1EhH7pHM with the following message: Error: Command failed with exit code 1: /app/node_modules/youtube-dl/bin/youtube-dl --dump-json -o users/admin/video/%(title)s.mp4 --write-info-json --print-json -f bestvideo+bestaudio --merge-output-format mp4 --write-thumbnail --proxy socks5://tor.proxy.local:9150 http://www.youtube.com/watch?v=92g1EhH7pHM

youtube doesnt like me and returns every request with a 429 error so i use a tor proxy
but even without the proxy i get error 1 and not 429

Tzahi12345 commented 3 years ago

This could simply be that youtube-dl is outdated. @vista-narvas I got the same error so at least it's reproduced. Once #234 is implemented, hopefully this will be fixed!

vista-narvas commented 3 years ago

it works but only without the proxy args if i use the proxy i get this error (the same error as before) it doesnt matter if i use youtube-dl or youtube-dlc without the proxy i would get banned in notime so i do need it

2020-11-02T22:57:23.970Z VERBOSE: youtube-dl args being used: -o,users/admin/video/%(title)s.mp4,--write-info-json,--print-json,-f,bestvideo+bestaudio,--merge-output-format,mp4,--write-thumbnail, --proxy socks5://tor.proxy.local:9150
2020-11-02T22:57:28.761Z ERROR: Error while retrieving info on video with URL https://www.youtube.com/watch?v=VAvljsMrY80 with the following message: Error: Command failed with exit code 1: /app/node_modules/youtube-dl/bin/youtube-dl --dump-json -o users/admin/video/%(title)s.mp4 --write-info-json --print-json -f bestvideo+bestaudio --merge-output-format mp4 --write-thumbnail --proxy socks5://tor.proxy.local:9150 http://www.youtube.com/watch?v=VAvljsMrY80

if i use the host name of the server for the proxy i get this error (with both youtube-dl and youtube-dlc) (this works when i use it on my pc)

2020-11-02T23:12:06.514Z VERBOSE: youtube-dl args being used: -o,users/admin/video/%(title)s.mp4,--write-info-json,--print-json,-f,bestvideo+bestaudio,--merge-output-format,mp4,--write-thumbnail,--proxy socks://server1:9150
2020-11-02T23:12:09.084Z ERROR: Error while retrieving info on video with URL https://www.youtube.com/watch?v=J---aiyznGQ with the following message: Error: Command failed with exit code 2: /app/node_modules/youtube-dl/bin/youtube-dl --dump-json -o users/admin/video/%(title)s.mp4 --write-info-json --print-json -f bestvideo+bestaudio --merge-output-format mp4 --write-thumbnail --proxy socks://server1:9150 http://www.youtube.com/watch?v=J---aiyznGQ

its a different error failed with exit code 2

youtube-dl on my pc is version 2020.09.20 and Python version 3.8.6 for YoutubeDL-Material i use the docker nightly tag

ghost commented 3 years ago

If you use SSL inspection in your network (e.g. Untangle's SSL Inspector) and it is configured to inspect youtube traffic, try disabling the SSL inspection or add a source IP address exception. This was the resolution in my case.

vista-narvas commented 3 years ago

i think i fixed it by adding HTTP_PROXY to the environment section in my docker-compose file

    environment:
      HTTP_PROXY: "socks://tor.proxy.local:9150"
      HTTPS_PROXY: "socks://tor.proxy.local:9150"
      FTP_PROXY: "socks://tor.proxy.local:9150"

i have no way to verify if its using the proxy or normal connection but when i disable the proxy i get an error so i think it works

Tzahi12345 commented 3 years ago

@vista-narvas Interesting.. if that's the case, then the recommended method to avoid 429 errors via proxy should be to do it in the docker-compose. Good to know!

JasonGhent commented 3 years ago

@vista-narvas I just started looking into this last night and saw similar behavior when trying to use a SOCKS proxy.

You may note in the code that there are a few places that youtubedl.exec is being called, and none of them pass in (or allow a youtubedl-material end-user to pass in) additional options to the third argument parameter, nor do they allow a way to append the --proxy argument to the downloadConfig array second argument parameter.

I think updating the service to allow this via an environment variable would be trivial, so if no one else does this before I get a chance, I'll try to put together a PR sometime later this week.

Tzahi12345 commented 3 years ago

@JasonGhent

You may note in the code that there are a few places that youtubedl.exec is being called, and none of them pass in (or allow a youtubedl-material end-user to pass in) additional options to the third argument parameter, nor do they allow a way to append the --proxy argument to the downloadConfig array second argument parameter.

You're half-right, in that with the exec method there's no dedicated parameter for proxy. Unfortunately, you can't simply pass in the proxy as it's described in the node-youtube-dl docs, because that extra param only applies to the non-exec method, and we only use the exec method.

Note that the downloadConfig is an array of args, so the current method of using a proxy is to add the proxy arg to the downloadConfig (apparently this doesn't work 100%?). This is done using the global custom args in the Downloader tab in the settings (these apply to downloads on the home page, if you want to pass custom args into subscriptions you need to use the subscription-specific custom args).

This is a little bit confusing, so if you need clarification I'll be happy to help (head over to the Discussions and open a thread there).

JasonGhent commented 3 years ago

@Tzahi12345 You're right that this can be addressed as a custom arg through the UI. I was looking at it from the perspective of a way to do it with environment variables. Mea culpa.

I did have an issue, though:

0000-00-00T00:00:00.000Z ERROR: Error while retrieving info on video with URL https://www.youtube.com/watch?v=xxxxxxxx with the following message: Error: Command failed with exit code 2: /app/node_modules/youtube-dl/bin/youtube-dl --dump-json -o video/%(title)s.mp4 --write-info-json --print-json -fbestvideo+bestaudio --merge-output-format mp4 --write-thumbnail --proxy http://127.0.0.1:1234 http://www.youtube.com/watch?v=xxxxxxxx

¯_(ツ)_/¯

Tzahi12345 commented 3 years ago

@JasonGhent No problem! I think there's several ways to go about this problem, maybe I should add a dedicated setting for it since it seems used pretty often. Or I can just tell users to use the docker-compose with Docker, or custom args without Docker.

Regarding your error, it looks like the args might be badly formed, or is this just a copy-paste error: -fbestvideo+bestaudio?

It should instead be -f bestvideo+bestaudio