Error when downloading entire channel

[maggie2013]

I have used this to download an entire channel before with success youtube-dl -citvw ytuser:USER

I have tried now with a username that contains spaces so I used double quotes but am getting an error. Can someone recommend a workaround or is this is bug?

% youtube-dl -citvw ytuser:"Triumph, the Insult Comic Dog"
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['-citvw', 'ytuser:Triumph, the Insult Comic Dog']
[debug] youtube-dl version 2013.11.07
[debug] Python version 3.3.2 - Linux-3.11.6-1-x86_64-with-arch
[debug] Proxy map: {}
[youtube:user] Triumph: Downloading video ids from 1 to 51
ERROR: Unable to download webpage: HTTP Error 404: Not Found; please report this issue on https://yt-dl.org/bug . Be sure to call youtube-dl with the --verbose flag and include its complete output. Make sure you are using the latest version; type  youtube-dl -U  to update.
  File "/usr/lib/python3.3/site-packages/youtube_dl/extractor/common.py", line 155, in _request_webpage
    return compat_urllib_request.urlopen(url_or_request)
  File "/usr/lib/python3.3/urllib/request.py", line 156, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.3/urllib/request.py", line 475, in open
    response = meth(req, response)
  File "/usr/lib/python3.3/urllib/request.py", line 587, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.3/urllib/request.py", line 513, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.3/urllib/request.py", line 447, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.3/urllib/request.py", line 595, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)

[jaimeMF]

Could you post a link to the user profile or some of its video?

[maggie2013]

http://www.youtube.com/channel/HCXxVDS49YA7g/videos

[jaimeMF]

Since you have the url and it's not a real user, but an automatically generated channel, the best option is to just run youtube-dl http://www.youtube.com/channel/HCXxVDS49YA7g.

[maggie2013]

Thank worked great. Maybe add a section in the manpage that shows both operations?

[jaimeMF]

Well, the synopsis is quite clear, just pass the url to youtube-dl and it will do all the work for you. That's what I personally did when I started using the program and that's what I think we should recommend.

[ghost]

Hi. This doesn't currently work right now...apparently. I've tried:

$ youtube-dl --extract-audio -citw --audio-format mp3 --user-agent "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:26.0) Gecko/20100101 Firefox/26.0" ytuser:ManiacSynth
[download] Downloading playlist: ManiacSynth
[youtube:user] ManiacSynth: Downloading video ids from 1 to 51
ERROR: Unable to download webpage: <urlopen error [Errno 111] Connection refused>

and:

$ youtube-dl --extract-audio -citw --audio-format mp3 --user-agent "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:26.0) Gecko/20100101 Firefox/26.0" https://www.youtube.com/user/ManiacSynth/
[download] Downloading playlist: ManiacSynth
[youtube:user] ManiacSynth: Downloading video ids from 1 to 51
ERROR: Unable to download webpage: <urlopen error [Errno 111] Connection refused>

and a few more variants like: https://www.youtube.com/user/ManiacSynth https://www.youtube.com/user/ManiacSynth/videos with the same results.

I'm running the current youtube-dl -U version which is: 2014.03.17

Thank you in advance for any help.

EDIT: oh i forgot to add -v, here's one:

$ youtube-dl --extract-audio -citw --audio-format mp3 --user-agent "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:26.0) Gecko/20100101 Firefox/26.0" https://www.youtube.com/user/ManiacSynth/videos -v
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['--extract-audio', '-citw', '--audio-format', 'mp3', '--user-agent', 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:26.0) Gecko/20100101 Firefox/26.0', 'https://www.youtube.com/user/ManiacSynth/videos', '-v']
[debug] Encodings: locale 'UTF-8', fs 'UTF-8', out 'UTF-8', pref: 'UTF-8'
[debug] youtube-dl version 2014.03.17
[debug] Python version 2.7.4 - Linux-3.8.0-30-generic-x86_64-with-LinuxMint-15-olivia
[debug] Proxy map: {}
[download] Downloading playlist: ManiacSynth
[youtube:user] ManiacSynth: Downloading video ids from 1 to 51
ERROR: Unable to download webpage: <urlopen error [Errno 111] Connection refused>
  File "/usr/bin/youtube-dl/youtube_dl/extractor/common.py", line 194, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1191, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python2.7/urllib2.py", line 404, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 422, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/bin/youtube-dl/youtube_dl/utils.py", line 579, in https_open
    return self.do_open(HTTPSConnectionV3, req)
  File "/usr/lib/python2.7/urllib2.py", line 1184, in do_open
    raise URLError(err)

[phihag]

@emanueLczirai This works fine for me. Are you certain that you are not behind some kind of firewall? The error you're getting is a network error. Can you post the output you get for

youtube-dl -v --proxy http://l.phihag.de:3120 https://www.youtube.com/user/ManiacSynth/videos

[ghost]

omg, you're completely right! That works! I am behind a firewall but I think the problem is that I've added some hosts entries blocking some google sites (like google plus) and one of them is clearly needed for this operation (although getting individual videos and videos from playlists works fine, with youtube-dl also with chromium but for example comments never load on videos due to this hosts thing)

Thanks so much! @phihag

How would I go about finding out which hostnames youtube-dl tries to access? EDIT: nvm, I'm trying to figure out how to tell dnsmasq to log entries right now. EDIT2: oh i didn't think this through:) dnsmasq wouldn't catch these, so I'll just remove hosts temporarily just to make it catch them. EDIT3: found it, the hostname gdata.youtube.com was 127.0.0.1 for me :) and it was needed in order for this to run. Fixed and working as we speak.

here's the output, just in case in matters:

$ youtube-dl -v --proxy http://l.phihag.de:3120 https://www.youtube.com/user/ManiacSynth/videos 
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['-v', '--proxy', 'http://l.phihag.de:3120', 'https://www.youtube.com/user/ManiacSynth/videos']
[debug] Encodings: locale 'UTF-8', fs 'UTF-8', out 'UTF-8', pref: 'UTF-8'
[debug] youtube-dl version 2014.03.17
[debug] Python version 2.7.4 - Linux-3.8.0-30-generic-x86_64-with-LinuxMint-15-olivia
[debug] Proxy map: {u'http': 'http://l.phihag.de:3120', u'https': 'http://l.phihag.de:3120'}
[download] Downloading playlist: ManiacSynth
[youtube:user] ManiacSynth: Downloading video ids from 1 to 51
[youtube:user] ManiacSynth: Downloading video ids from 51 to 101
[youtube:user] ManiacSynth: Downloading video ids from 101 to 151
[youtube:user] ManiacSynth: Downloading video ids from 151 to 201
[youtube:user] ManiacSynth: Downloading video ids from 201 to 251
[youtube:user] ManiacSynth: Downloading video ids from 251 to 301
^C
ERROR: Interrupted by user

I am currently trying without the -v and it works...

[ghost]

I needed to allow gdata.youtube.com in /etc/hosts (was previously 127.0.0.1 by me) Thanks again!

EDIT: for reference, this is the command that I ended up using:

youtube-dl --extract-audio -ctw --audio-format mp3 --user-agent "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:26.0) Gecko/20100101 Firefox/26.0" -k --download-archive ./alldownloadedIDs https://www.youtube.com/user/ManiacSynth 

works well when Ctrl+C -ing while download or [avconv] are in progress. (avoided the use of -i due to Ctrl+C -ing while in [avconv] would fail as expected but the video would not be processed on the next run, because it's ID would be cached in the specified archive)

[phihag]

Great. You can see the host names after Host: when you pass in --print-traffic. As to your command: Do you really need to pass in -t and --user-agent? These options are harmless at best.

[ghost]

Awesome, I remember seeing --print-traffic but completely forgot about it! Great to know! the user agent remained from when I was attempting to work around the connection refused the first time; I was thinking maybe they do that automatically based on user agent, but on further thought that is not how it works, unless maybe the page checking the user agent purposefully redirected me to a hostname which always refuses connections (so unlikely!) - that's what happens when you(aka me) try to superficially deduce why things happen :)) the deeper I go the more I realize how wrong I was for thinking that was the case in the first place.

Looks like with -t or without, I get the same result (hence why you said harmless I guess haha), (example BBRAINZ - リフレクション-q8pTerMgmR4.mp3 I especially like the videoID in the name) but maybe the defaults can change in the future and I like to specify my defaults where possible. I'll keep passing -t because feels good being in accordance with my beliefs of explicitly stating those (evil)implicit defaults (generally speaking) :D

Peaces.

[ghost]

I noticed that -w flag (aka --no-overwrites) doesn't work for the [avconv] part; in other words if the .mp3 file exists (or not) it will still be created anew.

-w, --no-overwrites
           Do no overwrite already existing files.

ie. this command youtube-dl https://www.youtube.com/watch?v=L3j7qwyNRG8 --extract-audio -w -k --audio-format mp3 will recreate the .mp3 on each run

I guess this is meant only for downloading files. Maybe the doc could say so? Not sure if I should raise an issue of this or not... I figure that the existing .mp3 file could be incomplete (due to Ctrl+C maybe? tested, confirmed that Ctrl+C does leave in complete .mp3) and thus it would be a tough call whether to recreate it or not, maybe also depending on -c flag (aka --continue but that one's for sure only for downloaded files anyway Resume partially downloaded files.) So I guess recreating the mp3 is not a bad idea because you never know if it was left incomplete... but maybe just stating that -w only applies to downloaded files and not converted files would be neat (although I haven't read the entire man page, maybe there's mention of this)

[ghost]

Ah nevermind that, I was looking only inside the man page (aka man youtube-dl) which doesn't list --no-post-overwrites (found by youtube-dl --help|less) possibly because, maybe, the man page is a leftover of a previous version of youtube-dl ? and since I used youtube-dl -U to update, that would explain it. EDIT2: yep, man page says: youtube-dl 2012-11-17 YOUTUBE-DL(1)

EDIT: so adding --no-post-overwrites will not overwrite existing .mp3 files (tested) ie. [youtube] Post-process file Kirk Gadget - Hot Pursuit-L3j7qwyNRG8.mp3 exists, skipping

[Xpr3ss]

Hi @maggie2013 since that channel's been taken down, any chance you would have his videos saved somewhere?