search

yt-dlp / yt-dlp

A feature-rich command-line audio/video downloader
https://discord.gg/H5MNcFW63r
The Unlicense
86.81k stars 6.77k forks source link

yt-dlp Failing to Download Youtube Videos #4349

Closed richardstevenhack closed 2 years ago

richardstevenhack commented 2 years ago

Checklist

Please make sure the question is worded well enough to be understood

I am running on openSUSE Leap 15.3 Linux. My python is 3.6.15.

In an attempt to implement more options to avoid Youtube throttling, I began experimenting with the command line functions. I was using Media-Downloader to download fairly large numbers of videos per day - around 60-70. To avoid Youtube throttling, I began using the --chunk-size, and --sleep-interval options. I had used the sleep interval option before OK.

Also I saw that Media-Downloader had added the .200s. option to solve the problem of too long file names.

Media-Downloader also defaulted to using the id string, which I didn't want, so I edited that out of the default file output format. I just wanted file name and extension.

Anyway, I got nothing but download failures. So to eliminate Media-Downloader as the issue, I began using the same command line output on the actual command line. I am still getting a failure to download (see verbose output below).

I am assuming there is no bug here, but that I am doing something seriously wrong.

The command line I'm using is this: yt-dlp -vU --http-chunk-size 2M --sleep-interval 10 --newline --ignore-config --no-playlist -o "%(title).200s.%(ext)s" "https://www.youtube.com/watch?v=HSE9RmEyE9E"

The first error I get is:

[download] Destination: Safe Money Trasnfer Procedures - Prevent wire fraud.f248.webm ERROR: unable to download video data: [Errno 2] No such file or directory: 'Safe Money Trasnfer Procedures - Prevent wire fraud.f248.webm.part-Frag1.part'

Then it reports a traceback with the primary error being: socket.timeout: The read operation timed out Then there is a further traceback because in the handling of that exception, another exception occurred which is: FileNotFoundError: [Errno 2] No such file or directory: 'Safe Money Trasnfer Procedures - Prevent wire fraud.f248.webm.part-Frag1.part' For some reason it seems to be parsing the output file name as the fragment part of the file, or so it seems to me.

I can't make head or tails of the traceback, so that's no help to me. I assume that there is something wrong with the command line I'm using, but after hours of reviewing the command line guidance document, I can't see it.

Thank you for your attention.

Provide verbose output that clearly demonstrates the problem

Complete Verbose Output

rhack@localhost:/Data2/Work> yt-dlp -vU --http-chunk-size 2M --sleep-interval 10 --newline --ignore-config --no-playlist -o "%(title).200s.%(ext)s" "https://www.youtube.com/watch?v=HSE9RmEyE9E"
[debug] Command-line config: ['-vU', '--http-chunk-size', '2M', '--sleep-interval', '10', '--newline', '--ignore-config', '--no-playlist', '-o', '%(title).200s.%(ext)s', 'https://www.youtube.com/watch?v=HSE9RmEyE9E']
DeprecationWarning: Support for Python version 3.6 has been deprecated. See  https://github.com/yt-dlp/yt-dlp/issues/3764  for more details. You will recieve only one more update on this version! Please update to Python 3.7 or above
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out UTF-8, error UTF-8, screen UTF-8
[debug] yt-dlp version 2022.06.29 [9d339c4] (zip)
[debug] Python 3.6.15 (CPython 64bit) - Linux-5.3.18-150300.59.76-preempt-x86_64-with-glibc2.3.4 (glibc 2.3.4)
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] exe versions: ffmpeg 3.4.9, ffprobe 3.4.9
[debug] Optional libraries: brotli-None, certifi-2018.01.18, sqlite3-2.6.0
[debug] Proxy map: {}
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
[debug] Downloading _update_spec from https://github.com/yt-dlp/yt-dlp/releases/download/2022.06.29/_update_spec
Latest version: 2022.06.29, Current version: 2022.06.29
yt-dlp is up to date (2022.06.29)
[debug] [youtube] Extracting URL: https://www.youtube.com/watch?v=HSE9RmEyE9E
[youtube] HSE9RmEyE9E: Downloading webpage
[youtube] HSE9RmEyE9E: Downloading android player API JSON
[youtube] HSE9RmEyE9E: Downloading MPD manifest
[youtube] HSE9RmEyE9E: Downloading MPD manifest
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, codec:vp9.2, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), acodec, lang, proto, filesize, fs_approx, tbr, vbr, abr, asr, vext, aext, hasaud, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] HSE9RmEyE9E: Downloading 1 format(s): 248+251
[debug] Invoking dashsegments downloader on "https://manifest.googlevideo.com/api/manifest/dash/expire/1657788618/ei/aoTPYs3tG4PAkwbkzYHICQ/ip/2601%3A646%3A300%3A6860%3Ac6e3%3A3562%3A7727%3A918a/id/1d213d46613213d1/source/youtube/requiressl/yes/playback_host/rr2---sn-o097znss.googlevideo.com/mh/nJ/mm/31%2C26/mn/sn-o097znss%2Csn-a5mekn6k/ms/au%2Conr/mv/m/mvi/2/pl/38/tx/24027699/txs/24027699%2C24027700%2C24027701%2C24027702%2C24067854/hfr/all/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cwebm2_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm2_sd_hd_clear/initcwndbps/1911250/spc/lT-KhhGg6ohOuIQ3vi0C-PjK1x7SKkE/vprv/1/mt/1657766694/fvip/3/keepalive/yes/fexp/24001373%2C24007246/itag/0/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Ctx%2Ctxs%2Chfr%2Cas%2Cspc%2Cvprv%2Citag/sig/AOq0QJ8wRgIhAOdzurROoF-enAL775cMcXjVlXyyLoy10YJSAR0TUtAUAiEAysFSmPRQL81MC84gHf1MK3V7jVfa0kSK4pdxfBfF6ag%3D/lsparams/playback_host%2Cmh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps/lsig/AG3C_xAwRgIhAKdSdMKWi7g4AjUeCb02MYKhTzde7RgS1QbpMmVDtUE9AiEAp8KAgxEmNze1P1XoXIEYZdd1OV_vaGY0_Z-5K4IXRrM%3D"
[download] Sleeping 10.00 seconds ...
[dashsegments] Total fragments: 84
[download] Destination: Safe Money Trasnfer Procedures - Prevent wire fraud.f248.webm
ERROR: unable to download video data: [Errno 2] No such file or directory: 'Safe Money Trasnfer Procedures - Prevent wire fraud.f248.webm.part-Frag1.part'
Traceback (most recent call last):
  File "/usr/local/bin/yt-dlp/yt_dlp/downloader/http.py", line 253, in download
    data_block = ctx.data.read(block_size if not is_test else min(block_size, data_len - byte_counter))
  File "/usr/lib64/python3.6/http/client.py", line 467, in read
    n = self.readinto(b)
  File "/usr/lib64/python3.6/http/client.py", line 511, in readinto
    n = self.fp.readinto(b)
  File "/usr/lib64/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib64/python3.6/ssl.py", line 1012, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib64/python3.6/ssl.py", line 874, in read
    return self._sslobj.read(len, buffer)
  File "/usr/lib64/python3.6/ssl.py", line 631, in read
    v = self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 3111, in process_info
    partial_success, real_download = self.dl(fname, new_info)
  File "/usr/local/bin/yt-dlp/yt_dlp/YoutubeDL.py", line 2829, in dl
    return fd.download(name, new_info, subtitle)
  File "/usr/local/bin/yt-dlp/yt_dlp/downloader/common.py", line 444, in download
    ret = self.real_download(filename, info_dict)
  File "/usr/local/bin/yt-dlp/yt_dlp/downloader/dash.py", line 54, in real_download
    return self.download_and_append_fragments_multiple(*args)
  File "/usr/local/bin/yt-dlp/yt_dlp/downloader/fragment.py", line 372, in download_and_append_fragments_multiple
    return self.download_and_append_fragments(*args[0], pack_func=pack_func, finish_func=finish_func)
  File "/usr/local/bin/yt-dlp/yt_dlp/downloader/fragment.py", line 524, in download_and_append_fragments
    download_fragment(fragment, ctx)
  File "/usr/local/bin/yt-dlp/yt_dlp/downloader/fragment.py", line 459, in download_fragment
    if self._download_fragment(ctx, fragment['url'], info_dict, headers):
  File "/usr/local/bin/yt-dlp/yt_dlp/downloader/fragment.py", line 125, in _download_fragment
    success, _ = ctx['dl'].download(fragment_filename, fragment_info_dict)
  File "/usr/local/bin/yt-dlp/yt_dlp/downloader/common.py", line 444, in download
    ret = self.real_download(filename, info_dict)
  File "/usr/local/bin/yt-dlp/yt_dlp/downloader/http.py", line 373, in real_download
    return download()
  File "/usr/local/bin/yt-dlp/yt_dlp/downloader/http.py", line 255, in download
    retry(err)
  File "/usr/local/bin/yt-dlp/yt_dlp/downloader/http.py", line 247, in retry
    ctx.resume_len = byte_counter if to_stdout else os.path.getsize(encodeFilename(ctx.tmpfilename))
  File "/usr/lib64/python3.6/genericpath.py", line 50, in getsize
    return os.stat(filename).st_size
FileNotFoundError: [Errno 2] No such file or directory: 'Safe Money Trasnfer Procedures - Prevent wire fraud.f248.webm.part-Frag1.part'
coletdjnz commented 2 years ago

try --check-formats

The youtube server you are connecting to may be having issues with f248

edit: remove --http-chunk-size 2M, that seems to be be causing it my end :thinking:

richardstevenhack commented 2 years ago

I just removed both the chunk and sleep interval and the file downloaded fine, so you may be right. I'll try it again with the sleep interval put back in.

richardstevenhack commented 2 years ago

Well, I'm trying - but Youtube is throttling this down to 49-50K speed, so it will be another 13 minutes before I know. I went on my NordVPN to see if I can evade that, but it's not helping. I think I'll try another video.

richardstevenhack commented 2 years ago

Well, that didn't work. Youtube will take two hours to download the new file. I'm going to assume that everything works but the chunk option.

Now the question is why? I used the chunk option because in a discussion of throttling issues, it was noted that Youtube had changed from 10M chunks to 2M chunks. Is the chunk option still useful?

coletdjnz commented 2 years ago

since this is dashsegments try bumping up -N, or try another format that isn't DASH (--extractor-args youtube:skip=dash)

this video is slow for me too (~1MB/s). I don't think it is throttling, rather it is just not that popular so it's probably not in hot cache.

coletdjnz commented 2 years ago

Now the question is why? I used the chunk option because in a discussion of throttling issues, it was noted that Youtube had changed from 10M chunks to 2M chunks. Is the chunk option still useful?

I think this is to do with it being a dashsegments video rather than normal http video. The HTTP chunk size issue (and option) only applies to the latter. However it looks like if you try to force it with dashsegments YouTube doesn't like it :thinking: .

(someone correct me if I'm wrong)

richardstevenhack commented 2 years ago

Oh, yeah, booping N up to 4 and now the video is coming down at full Comcast 100mbps speed. Let me try with another video.

richardstevenhack commented 2 years ago

The second 30MB video came down in five minutes, which is very slow. The speed reported varied from 20Kbs to the 5-6Mbps I usually get. There was a frequent pause of about 9 seconds consistently through every batch of fragments.

There was also a warning to the effect that the speed shown was only for one thread and "patches are welcome", so I don't know what that was about.

Anyway, it seems clear that the chunk option is not working for some reason and that yt-dlp doesn't like it if it's used with dashsegments videos.

richardstevenhack commented 2 years ago

OK, using the -N 5 option, I get full speed - but a delay of nine seconds between each batch of fragments. If someone could explain that, it might help.

Otherwise I guess this is the best I can get, so I'll have to live with it. I may start doing my daily downloads overnight so I don't have to babysit it.

pukkandan commented 2 years ago

Why do you think this issue is a good place to report your problem? Open a new issue with relevant logs