Not downloading full channel.

[cjaustin1792]

Describe the bug Channel has 500+ videos, I have set it up to watch for new uploads and checked download full channel. Downloaded about 75 videos and then decided to stop, unsure how to trigger it to continue.

To Reproduce Maybe try pulling large channel?

Expected behavior Download all channel and maybe add force download button?

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: Ubuntu 18.04 headless remote server

Additional context Add any other context about the problem here.

[XiovV]

Seems to be an issue with youtube-dl and latest youtube changes. https://www.reddit.com/r/DataHoarder/comments/e9nqph/were_doing_a_youtube_archiving_project/

[AMJidovu]

For me it does not even add the channel to the download without giving an error, so there is no folder created and no video downloaded regardless of channel. The only error in the log is : {"level":"error","msg":"From GetMetadata(): exit status 1","time":"2019-12-17T08:44:18Z"}

[AMJidovu]

This is the full log from trying to download a full channel. I replaced the channel name I used with "user".

{"level":"info","msg":"server running on port 8080","time":"2019-12-18T04:00:53Z"}
{"level":"error","msg":"uploadCheckerPlaylists: GetCheckingInterval: empty target list","time":"2019-12-18T04:00:53Z"}
{"level":"error","msg":"uploadCheckerChannels: GetCheckingInterval: empty target list","time":"2019-12-18T04:00:53Z"}
{"level":"info","msg":"got this data: {channels}","time":"2019-12-18T04:03:42Z"}
{"level":"info","msg":"getting all channels from channels.json","time":"2019-12-18T04:03:42Z"}
{"level":"info","msg":"successfully read all channels","time":"2019-12-18T04:03:42Z"}
{"level":"info","msg":"[]","time":"2019-12-18T04:03:42Z"}
{"level":"info","msg":"received a request to add a target","time":"2019-12-18T04:04:05Z"}
{"level":"info","msg":"executing youtube-dl command: /usr/local/bin/youtube-dl -j --playlist-end 1 https://www.youtube.com/user/user","time":"2019-12-18T04:04:05Z"}
{"level":"error","msg":"From GetMetadata(): exit status 1","time":"2019-12-18T04:04:10Z"}
{"level":"info","msg":"DOWNLOAD: {https://www.youtube.com/user/user  Video And Audio   any [] 12-18-2019 04:04:10  Channel /channels/%(uploader)s/video/%(title)s.%(ext)s}","time":"2019-12-18T04:04:10Z"}
{"level":"info","msg":"downloading video","time":"2019-12-18T04:04:10Z"}
{"level":"info","msg":"{ youtube-dl --ignore-errors  bestvideo[ext=any] downloads/channels/%(uploader)s/video/%(title)s.%(ext)s https://www.youtube.com/user/user}","time":"2019-12-18T04:04:10Z"}
{"level":"info","msg":"executing youtube-dl command: /usr/local/bin/youtube-dl --ignore-errors  -f bestvideo[ext=any] -o downloads/channels/%(uploader)s/video/%(title)s.%(ext)s https://www.youtube.com/user/user","time":"2019-12-18T04:04:10Z"}
{"level":"error","msg":"uploadCheckerPlaylists: GetCheckingInterval: empty target list","time":"2019-12-18T04:05:53Z"}
{"level":"error","msg":"uploadCheckerChannels: strconv.Atoi: parsing \"\": invalid syntax","time":"2019-12-18T04:05:53Z"}
{"level":"error","msg":"exit status 1/usr/local/bin/youtube-dl --ignore-errors  -f bestvideo[ext=any] -o downloads/channels/%(uploader)s/video/%(title)s.%(ext)s https://www.youtube.com/user/user","time":"2019-12-18T04:06:41Z"}
{"level":"info","msg":"latest downloaded video id updated successfully","time":"2019-12-18T04:06:41Z"}
{"level":"info","msg":"[]","time":"2019-12-18T04:06:41Z"}
{"level":"info","msg":"returning response: {Error ERROR_DOWNLOADING There was an error downloading the target Download: DownloadVideo: exit status 1}","time":"2019-12-18T04:06:41Z"}
{"level":"info","msg":"got this data: {channels}","time":"2019-12-18T04:06:41Z"}
{"level":"info","msg":"getting all channels from channels.json","time":"2019-12-18T04:06:41Z"}
{"level":"info","msg":"successfully read all channels","time":"2019-12-18T04:06:41Z"}
{"level":"info","msg":"[{https://www.youtube.com/user/user  Video And Audio   any [] 12-18-2019 04:04:10  Channel /channels/%(uploader)s/video/%(title)s.%(ext)s}]","time":"2019-12-18T04:06:41Z"}

[XiovV]

Seems to be working fine for me, I just pushed 0.5.8 when I fixed #73, try updating a let me know if the issue still persists.

[AMJidovu]

This is the error I get now when I try to download a full channel using the latest version of the app with docker. There was an error getting channel metadata: From c.GetMetadata(): WARNING: unable to download video info webpage: HTTP Error 429: Too Many Requests WARNING: unable to download video info webpage: HTTP Error 429: Too Many Requests WARNING: unable to download video info webpage: HTTP Error 429: Too Many Requests WARNING: unable to download video info webpage: HTTP Error 429: Too Many Requests ERROR: M2w3NZzPwOM: YouTube said: Unable to extract video data

And this is the full log of different channel but same exact errors:

{"level":"info","msg":"received a request to add a target","time":"2019-12-18T17:13:00Z"}
{"level":"info","msg":"executing youtube-dl command: /usr/local/bin/youtube-dl -j --playlist-end 1 https://www.youtube.com/channel/UCZDfnUn74N0WeAPvMqTOrtA","time":"2019-12-18T17:13:00Z"}
{"level":"error","msg":"From GetMetadata(): exit status 1","time":"2019-12-18T17:13:03Z"}

{"level":"info","msg":"returning response: {Error ERROR_GETTING_METADATA There was an error getting channel metadata: From c.GetMetadata(): WARNING: unable to download video info webpage: HTTP Error 429: Too Many Requests\nWARNING: unable to download video info webpage: HTTP Error 429: Too Many Requests\nWARNING: unable to download video info webpage: HTTP Error 429: Too Many Requests\nWARNING: unable to download video info webpage: HTTP Error 429: Too Many Requests\nERROR: RDPQXL5_mqY: YouTube said: Unable to extract video data\n}","time":"2019-12-18T17:13:03Z"}

{"level":"info","msg":"got this data: {channels}","time":"2019-12-18T17:13:03Z"}
{"level":"info","msg":"getting all channels from channels.json","time":"2019-12-18T17:13:03Z"}
{"level":"info","msg":"successfully read all channels","time":"2019-12-18T17:13:03Z"}
{"level":"info","msg":"[]","time":"2019-12-18T17:13:03Z"}

[XiovV]

I see, this is a youtube-dl problem. I'll do some research to figure out if there's anything I can do to prevent this.

[AMJidovu]

Thanks, I appreciate the effort.

[AMJidovu]

I had an user reporting on the forum that it works for him as long as he is not using the option "any (recommended for now)" , maybe remove that option ? I guess if there is something that can be done to fix the HTTP error 429 will work. Some people reported that if they use "--cookies" option they no longer get 429 HTTP error.

This is his post:

"Looks like the gui adds "[ext=any]" which isn't a supported option for youtube-dl

If you remove it from the config it'll just stick in "[ext=]" which of course also breaks

"Any" should just leave out the -f option as youtube-dl defaults to "-f bestvideo+bestaudio/best" which will try to get the best of both and convert them or grab the best if it's missing conversion tools"

[XiovV]

I fixed that in the dev branch, I just have to make a few more changes and I'll merge it into master.

[AMJidovu]

I had a look over the changes, so far looks good. Though I wish youtube-dl will do something about the HTTP error 429 which is basically a temp ban from youtube, which apparently no one knows exactly how youtube detects and handle those bans so they can be avoided.