search

ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
132.36k stars 10.04k forks source link

[Youtube] Get ERROR: giving up after 0 fragment retries (last nightly version) #32893

Closed DmytroUsenko closed 3 months ago

DmytroUsenko commented 3 months ago

Verbose log

DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): www.youtube.com:443 DEBUG:urllib3.connectionpool:https://www.youtube.com:443 "GET /watch?v=_OKAwz2MsJs HTTP/1.1" 200 None [debug] Encodings: locale cp1251, fs utf-8, out utf-8, pref cp1251 [debug] youtube-dl version 2024.08.02 [71223bff3] [debug] This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl. [debug] For support, visit the main site. [debug] Git HEAD: 1278be0 [debug] Python 3.9.10 (CPython AMD64 64bit) - Windows-10-10.0.19045-SP0 - OpenSSL 1.1.1m 14 Dec 2021 [debug] exe versions: ffmpeg 2023-01-12-git-fc263f073e-essentials_build-www.gyan.dev, ffprobe 2023-01-12-git-fc263f073e-essentials_build-www.gyan.dev [debug] Proxy map: {} [youtube] _OKAwz2MsJs: Downloading webpage [debug] [youtube] Decrypted nsig Qm536sut23muowSm => gPzfoGXIyPH5eg [debug] [youtube] Decrypted nsig y8r9VdSTeJPDq_6v => CV-R0tNLKCrfRw [debug] Invoking downloader on 'https://rr1---sn-585uxauxaxjvh-hw8l.googlevideo.com/videoplayback?expire=1722789855&ei=f1uvZuSAJ_CjvdIPo-G5uAo&ip=149.100.31.250&id=o-AAAGxGN9VxkYeMpSRs81kPCzbONTOrGmF8x0uraNOGFl&itag=135&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C271%2C278%2C313%2C394%2C395%2C396%2C397%2C398%2C399%2C400%2C401&source=youtube&requiressl=yes&xpc=EgVo2aDSNQ%3D%3D&mh=eD&mm=31%2C29&mn=sn-585uxauxaxjvh-hw8l%2Csn-h5q7kned&ms=au%2Crdu&mv=m&mvi=1&pl=21&initcwndbps=1710000&bui=AXc671IgcgyZgsxOhgZePMFXLDky3iVjgO1Od5g1l6RUx-WA4vPN7R8J_lkLbqxT3voVEjz1e7MpGqsf&spc=NO7bAeZtuJFJ6ceo0qpxyQy2Yhrcrn6HvpN-bkdNHKL4A9Z30NzWCyPWNuiR&vprv=1&svpuc=1&mime=video%2Fmp4&ns=UY2gwW6Ofrf9ldh1b7xBGi0Q&rqh=1&gir=yes&clen=8725720&dur=170.169&lmt=1721766646012129&mt=1722767837&fvip=5&keepalive=yes&c=WEB&sefc=1&txp=5532434&n=CV-R0tNLKCrfRw&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cxpc%2Cbui%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cns%2Crqh%2Cgir%2Cclen%2Cdur%2Clmt&sig=AJfQdSswRQIgUJda5-y-AO7qlh9UCBAGM8EZkTjGf1ldrwfRy3QjnQQCIQDOFXG5nsJJ7BDIVdJsIedFXJV1MaARmy5VDJQ5TIovEw%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AGtxev0wRQIhAMvFfD-S1-dmo2iy6hEn2IkKbC-4hasHZDrHgyhzhEB4AiBGYfPux3Y3VH6iMXIdPrJgREd8322H7fsZeWEbd6iRFA%3D%3D' [dashsegments] Total fragments: 1 [download] Destination: Temp\4db3dfbb-bdc3-404f-a1e7-c3a49aeb0938\Joker - Folie À Deux _ Official Trailer.f135.mp4 ERROR: giving up after 0 fragment retries

Description

Hey. Stuck with an issue that can't download files after last nightly update 2024.08.02 file https://www.youtube.com/watch?v=_OKAwz2MsJs pipeline: 1) The first run and second were successful 2) get an error after the third run 3) the same code on linux machine behaves the same 4) --rm-cache-dir or manually deleting the cache helps in most cases and I can download files (or other files from youtube) but the 2d or 3d attempt again brings up the issue. use in code:

...
        options = {
            "format": "bestvideo[height <=? 480][ext = mp4]+bestaudio[ext = m4a] / best[ext = mp4]",
            "outtmpl": f"{temp_subfolder}/%(title)s.%(ext)s",
            'nocheckcertificate': True,
            'ignoreerrors': True,
            "ffmpeg_location": r"D:\ffmpeg\bin",
            "verbose": True,
        }
...

                    with youtube_dl.YoutubeDL(options) as ydl:
                        ydl.download([url])
dirkf commented 3 months ago

I can't repro here. The reason for giving up after 0 retries doesn't seem to be logged; maybe inserting a pdb.set_trace() (say, youtube_dl/downloader/dash.py:21) would help.

I find that --external-downloader ffmpeg is always (?) giving 403, but this PC has a rather ancient ffmpeg. You might also try any available external downloaders.

@dant4ick, do feel free to add any substantive comment. We are all rolling our eyes here.

DmytroUsenko commented 3 months ago

I can't repro here. The reason for giving up after 0 retries doesn't seem to be logged; maybe inserting a pdb.set_trace() (say, youtube_dl/downloader/dash.py:21) would help.

I find that --external-downloader ffmpeg is always (?) giving 403, but this PC has a rather ancient ffmpeg. You might also try any available external downloaders.

@dant4ick, do feel free to add any substantive comment. We are all rolling our eyes here.

many thx @dirkf yeah. this is traceback 

[dashsegments] Total fragments: 1
[download] Destination: Temp\702f6ca2-ab5a-434c-a7c1-8ce4172e1574\Joker - Folie À Deux _ Official Trailer.m4a.f140-drc.m4a
ERROR: giving up after 0 fragment retries
  File "C:\Users\Dimitri\AppData\Local\Programs\PyCharm Professional\plugins\python\helpers\pydev\pydevd.py", line 2252, in <module>
    main()
  File "C:\Users\Dimitri\AppData\Local\Programs\PyCharm Professional\plugins\python\helpers\pydev\pydevd.py", line 2234, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "C:\Users\Dimitri\AppData\Local\Programs\PyCharm Professional\plugins\python\helpers\pydev\pydevd.py", line 1544, in run
    return self._exec(is_module, entry_point_fn, module_name, file, globals, locals)
  File "C:\Users\Dimitri\AppData\Local\Programs\PyCharm Professional\plugins\python\helpers\pydev\pydevd.py", line 1551, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "C:\Users\Dimitri\AppData\Local\Programs\PyCharm Professional\plugins\python\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "D:\Test\Docs\yp.py", line 82, in <module>
    parse_youtube_link(youtube_link)
  File "D:\Test\Docs\yp.py", line 58, in parse_youtube_link
    ydl.download([url])
  File "D:\Test\venv\lib\site-packages\youtube_dl\YoutubeDL.py", line 2257, in download
    res = self.extract_info(
  File "D:\Test\venv\lib\site-packages\youtube_dl\YoutubeDL.py", line 872, in extract_info
    return self.__extract_info(url, ie, download, extra_info, process)
  File "D:\Test\venv\lib\site-packages\youtube_dl\YoutubeDL.py", line 879, in wrapper
    return func(self, *args, **kwargs)
  File "D:\Test\venv\lib\site-packages\youtube_dl\YoutubeDL.py", line 986, in __extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "D:\Test\venv\lib\site-packages\youtube_dl\YoutubeDL.py", line 1020, in process_ie_result
    return self.process_video_result(ie_result, download=download)
  File "D:\Test\venv\lib\site-packages\youtube_dl\YoutubeDL.py", line 1864, in process_video_result
    self.process_info(new_info)
  File "D:\Test\venv\lib\site-packages\youtube_dl\YoutubeDL.py", line 142, in wrapper
    return func(self, *args, **kwargs)
  File "D:\Test\venv\lib\site-packages\youtube_dl\YoutubeDL.py", line 2156, in process_info
    partial_success = dl(fname, new_info)
  File "D:\Test\venv\lib\site-packages\youtube_dl\YoutubeDL.py", line 2092, in dl
    return fd.download(name, new_info)
  File "D:\Test\venv\lib\site-packages\youtube_dl\downloader\common.py", line 380, in download
    return self.real_download(filename, info_dict)
  File "D:\Test\venv\lib\site-packages\youtube_dl\downloader\dash.py", line 78, in real_download
    self.report_error('giving up after %s fragment retries' % count)
  File "D:\Test\venv\lib\site-packages\youtube_dl\downloader\common.py", line 175, in report_error
    self.ydl.report_error(*args, **kargs)
  File "D:\Test\venv\lib\site-packages\youtube_dl\YoutubeDL.py", line 682, in report_error
    self.trouble(*args, **kwargs)
  File "D:\Test\venv\lib\site-packages\youtube_dl\YoutubeDL.py", line 634, in trouble
    tb_data = traceback.format_list(traceback.extract_stack())
dirkf commented 3 months ago

You could try to see what exception is being raised on the download by either tracing in the DASH downloader module that I identified above or inserting print() calls there and elsewhere. Presumably it gets to l.53:

                    success, frag_content = self._download_fragment(ctx, fragment_url, info_dict, headers)

That call (into fragment.py, same directory) is returning a falsy success (not raising DownloadError, since on the first fragment fatal is True). Looking at the called method, we see that it's actually False because its first return value can only be that or True. So the failure is a falsy return from, presumably, the HTTP downloader (http.py, same directory); the circumstances where that would happen are unclear to me.

DmytroUsenko commented 3 months ago

I found out that some fragment_url for example this one 

https://rr1---sn-585uxauxaxjvh-hw8l.googlevideo.com/videoplayback?expire=1722796824&ei=uHavZvrKMIG5mLAP6d-X0A4&ip=149.100.31.250&id=o-AN27AHuO_OYDC1ZPdZF62x1ckTi7EZShC7Y84MPLg97t&itag=135&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C271%2C278%2C313%2C394%2C395%2C396%2C397%2C398%2C399%2C400%2C401&source=youtube&requiressl=yes&xpc=EgVo2aDSNQ%3D%3D&mh=eD&mm=31%2C29&mn=sn-585uxauxaxjvh-hw8l%2Csn-h5qzen7l&ms=au%2Crdu&mv=m&mvi=1&pl=21&initcwndbps=1387500&bui=AXc671K11H8_xro4zuscFP71W_128wNGQnQhNi2ZcAB4HaomqnnJyF3XEl7stcIyfHA6H8mAxtzLNFvi&spc=NO7bARZouLLeXCwgfMYoPEYMP7B1RzPRoYTpVKKKX_jbGOmgShTkrOz3axgB&vprv=1&svpuc=1&mime=video%2Fmp4&ns=q8p4f_XeRBg1-wQJxabKHIgQ&rqh=1&gir=yes&clen=8725720&dur=170.169&lmt=1721766646012129&mt=1722774803&fvip=5&keepalive=yes&c=WEB&sefc=1&txp=5532434&n=7fGekQpLcgIl6A&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cxpc%2Cbui%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cns%2Crqh%2Cgir%2Cclen%2Cdur%2Clmt&sig=AJfQdSswRAIgONMja921ObtPfl1lPLQJyTHQAAqpNCgsTDlz44iv2yMCIF5LN3CWKDCQ5dY3Q48vKA4sbdbz34I3I_zovSzvVbHv&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AGtxev0wRgIhAP7k0--rdwfa6uSinx5ztnYNCTTsZfgA2JU4bGTY_fyEAiEAgQIsjCwP9Pene6B9DJacV0mBzU5kZedeibR2up7SICw%3D&range=0-8725720

has status denied. ctx here is 

{'filename': 'Temp\\57180b1b-4c2c-4b70-8606-d1ac808d74e2\\Joker - Folie À Deux _ Official Trailer.f135.mp4', 'total_frags': 1, 'live': False, 'tmpfilename': 'Temp\\57180b1b-4c2c-4b70-8606-d1ac808d74e2\\Joker - Folie À Deux _ Official Trailer.f135.mp4.part', 'fragment_index': 0, 'dl': <youtube_dl.downloader.fragment.HttpQuietDownloader object at 0x0000018527F94850>, 'dest_stream': <_io.BufferedWriter name='Temp\\57180b1b-4c2c-4b70-8606-d1ac808d74e2\\Joker - Folie À Deux _ Official Trailer.f135.mp4.part'>, 'complete_frags_downloaded_bytes': 0, 'started': 1722775233.7545826, 'fragment_started': 1722775233.7545826, 'prev_frag_downloaded_bytes': 0, 'frag_resume_len': 0}

so I catch error here :

except compat_urllib_error.HTTPError as err:
DmytroUsenko commented 3 months ago

I am curious if for some reason YouTube returns the URL with 403- is it possible to add other attempts to get the correct URL (fragment)?

dirkf commented 3 months ago

If so, why don't we see output from here? Is fragment_retries <= 1, say? Are you setting that value in the parameters passed to your download code?

                    if count < fragment_retries:
                        self.report_retry_fragment(err, frag_index, count + 1, fragment_retries)
                        continue
DmytroUsenko commented 3 months ago

because I have count an fragment_retries as 0 and jump to break PS. and in general - retries don't fix the issue. the problem that I got here dash.py:21

    def real_download(self, filename, info_dict):

info_dict with the URL for the fragment that already has 403. and nothing helps me. I think the validation could be good to set early when I just get a fragment URL

dirkf commented 3 months ago

Maybe letting the download retry will help. More likely, your problem is the poToken experiment.

dirkf commented 3 months ago

Continued in #32905.

DmytroUsenko commented 3 months ago

I think the issue exists. no? why do you close it?

GregoriusT commented 3 months ago

because it is continued in another issue post as stated right above your post, its merging issues into one centralized post.

the "as not planned" thing is github itself not giving devs the option to be less passive aggressive about closing reason

DmytroUsenko commented 3 months ago

@dirkf to avoid opening new issues - I saw a new type of error with the same code (just on linux platform). I made a new update (2024-08-07) and see this. any ideas on how to cope with it?

DEBUG:urllib3.connectionpool:https://www.youtube.com:443 "GET /watch?v=HL0TnWXLnzs HTTP/1.1" 200 None
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2024.08.07 [c5098961b]
[debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl.
[debug] ** For support, visit the main site.
[debug] Python 3.10.12 (CPython x86_64 64bit) - Linux-6.5.0-1023-aws-x86_64-with-glibc2.35 - OpenSSL 3.0.2 15 Mar 2022 - glibc 2.35
[debug] exe versions: ffmpeg 7.0.2, ffprobe 7.0.2
[debug] Proxy map: {}
[youtube] HL0TnWXLnzs: Downloading webpage
ERROR: Sign in to confirm you’re not a bot
This helps protect our community. Learn more
Traceback (most recent call last):
  File "/home/ubuntu/venv/lib/python3.10/site-packages/youtube_dl/YoutubeDL.py", line 879, in wrapper
    return func(self, *args, **kwargs)
  File "/home/ubuntu/venv/lib/python3.10/site-packages/youtube_dl/YoutubeDL.py", line 975, in __extract_info
    ie_result = ie.extract(url)
  File "/home/ubuntu/venv/lib/python3.10/site-packages/youtube_dl/extractor/common.py", line 571, in extract
    ie_result = self._real_extract(url)
  File "/home/ubuntu/venv/lib/python3.10/site-packages/youtube_dl/extractor/youtube.py", line 2248, in _real_extract
    raise ExtractorError(reason, expected=True)
youtube_dl.utils.ExtractorError: Sign in to confirm you’re not a bot
This helps protect our community. Learn more
dirkf commented 3 months ago

YT ban? Review https://github.com/yt-dlp/yt-dlp/issues/10128.

dirkf commented 3 months ago

Regarding "not planned",. it may not be clear if you aren't able to close issues, but the subhead under the Close as not planned menu item says Won't fix, can't repro, duplicate, stale; here it's item 3. I have a policy-like view that an issue can retrospectively become duplicate if a newer issue is more directly focused on the underlying problem, which is what "Continued in..." means.

DmytroUsenko commented 3 months ago

Regarding "not planned",. it may not be clear if you aren't able to close issues, but the subhead under the Close as not planned menu item says Won't fix, can't repro, duplicate, stale; here it's item 3. I have a policy-like view that an issue can retrospectively become duplicate if a newer issue is more directly focused on the underlying policy, which is what "Continued in..." means.

no problem. understood the idea.

DmytroUsenko commented 3 months ago

YT ban? Review yt-dlp/yt-dlp#10128.

yes. the same absolutely. this a fork of your youtube-dl seems? It's funny, but the contributor just closed the issue without any solution)

dirkf commented 3 months ago

It looks like you just have to wait for your IP to get unbanned, and/or use a different connection. Or conceivably, actually using YT in your browser (presumably, you'll have to sign in) will get you unbanned. Or maybe the solution to #32905 will change things.

Regarding yt-dlp, once upon a time, long before I was involved (and so this account is hearsay), yt-dl was forked to add features that the then maintainers wouldn't accept. That fork eventually died and was later resuscitated as yt-dlp. Although yt-dlp is not a fork in the GitHub sense, there is shared commit (ancient) history, a common program architecture and interfaces sufficiently similar that an extractor can be written to run in both programs. Basically, this program tries to remain compatible with existing platforms while yt-dlp targets only supported platforms (and so is able to deploy novel Python syntax and library features as well as a more extensive set of dependencies).

DmytroUsenko commented 3 months ago

many thx