[Globo] Still fails to complete downloads

Troyciv commented 6 years ago

[x] I've verified and I assure that I'm running youtube-dl 2018.06.02
[x] At least skimmed through the README, most notably the FAQ and BUGS sections
[x] Searched the bugtracker for similar issues including closed ones
[x] Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser

What is the purpose of your issue?

[x] Bug report (encountered problems with youtube-dl)

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add the -v flag to your command line you run youtube-dl with (youtube-dl -v <your command line>), copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

[debug] System config: []                                                                                                                      
[debug] User config: []                                                                                                                        
[debug] Custom config: []                                                                                                                      
[debug] Command-line args: ['-u', 'PRIVATE', '-p', 'PRIVATE', '-v', 'https://globoplay.globo.com/v/6783003/programa/']                         
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8                                                                               
[debug] youtube-dl version 2018.06.02                                                                                                          
[debug] Python version 3.6.3 (CPython) - Linux-4.13.0-43-generic-x86_64-with-Ubuntu-17.10-artful
[debug] exe versions: ffmpeg 3.3.4-2, ffprobe 3.3.4-2, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {}

Description of your issue, suggested solution and other information

The download starts as expected, but aborts after ~20min and I need to <ctrl-c> and run it again to resume the download. I need to repeat that process untill the download is finished.

This happens with the -u 'username' -p 'password' option as well as the --cookies file.txt option


[debug] System config: []                                                                                                                      
[debug] User config: []                                                                                                                        
[debug] Custom config: []                                                                                                                      
[debug] Command-line args: ['-u', 'PRIVATE', '-p', 'PRIVATE', '-v', 'https://globoplay.globo.com/v/6783003/programa/']                         
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8                                                                               
[debug] youtube-dl version 2018.06.02                                                                                                          
[debug] Python version 3.6.3 (CPython) - Linux-4.13.0-43-generic-x86_64-with-Ubuntu-17.10-artful
[debug] exe versions: ffmpeg 3.3.4-2, ffprobe 3.3.4-2, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {}
[Globo] Downloading JSON metadata
[Globo] 6783003: Downloading JSON metadata
[Globo] 6783003: Downloading security hash for 6783003_90_720_collection
[Globo] 6783003_90_720_collection: Downloading m3u8 information
[Globo] 6783003: Downloading security hash for 6783003_90_720_l_collection
[Globo] 6783003_90_720_l_collection: Downloading m3u8 information
[Globo] 6783003: Downloading security hash for 6783003_360_1080_collection
[Globo] 6783003_360_1080_collection: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://vodstreaming04a.video.globo.com/r360_1080/v86/dd/d0/de/6783003_de307a1559004b8b7228c50c245e78b68d4e097d
/6783003-manifest.smil.ism/6783003-manifest.smil-audio_por_1=128000-video_por=13467382.m3u8?hls_client_manifest_version=4'
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 246
[download] Destination: Jornal Nacional - Íntegra 02 Junho 2018-6783003.mp4
[download]  32.9% of ~3.97GiB at  1.07MiB/s ETA 41:05[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 82 (attempt
 1 of 10)...
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 82 (attempt 2 of 10)...
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 82 (attempt 3 of 10)...
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 82 (attempt 4 of 10)...
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 82 (attempt 5 of 10)...
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 82 (attempt 6 of 10)...
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 82 (attempt 7 of 10)...
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 82 (attempt 8 of 10)...
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 82 (attempt 9 of 10)...
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 82 (attempt 10 of 10)...
[download] Skipping fragment 82...
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 83 (attempt 1 of 10)...
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 83 (attempt 2 of 10)...
[...]
^C                                 
ERROR: Interrupted by user

ghost commented 6 years ago

@Troyciv still fail here too with

sudo youtube-dl -U

version

youtube-dl is up-to-date (2018.06.04)

Troyciv commented 6 years ago

I timed it a few times and it seems the error always occurs after 20min. maybe the connection times out or something?

here is a traceback:


[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 376 (attempt 2 of 3)...
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 376 (attempt 3 of 3)...
ERROR: giving up after 3 fragment retries                    
  File "/home/user/.local/bin/youtube-dl", line 11, in <module>
    sys.exit(main())          
  File "/home/user/.local/lib/python3.6/site-packages/youtube_dl/__init__.py", line 472, in main
    _real_main(argv)          
  File "/home/user/.local/lib/python3.6/site-packages/youtube_dl/__init__.py", line 462, in _real_main
    retcode = ydl.download(all_urls)                         
  File "/home/user/.local/lib/python3.6/site-packages/youtube_dl/YoutubeDL.py", line 2001, in download
    url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "/home/user/.local/lib/python3.6/site-packages/youtube_dl/YoutubeDL.py", line 803, in extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "/home/user/.local/lib/python3.6/site-packages/youtube_dl/YoutubeDL.py", line 857, in process_ie_result
    return self.process_video_result(ie_result, download=download)
  File "/home/user/.local/lib/python3.6/site-packages/youtube_dl/YoutubeDL.py", line 1635, in process_video_result
    self.process_info(new_info)                              
  File "/home/user/.local/lib/python3.6/site-packages/youtube_dl/YoutubeDL.py", line 1908, in process_info
    success = dl(filename, info_dict)                        
  File "/home/user/.local/lib/python3.6/site-packages/youtube_dl/YoutubeDL.py", line 1847, in dl
    return fd.download(name, info)                           
  File "/home/user/.local/lib/python3.6/site-packages/youtube_dl/downloader/common.py", line 364, in download
    return self.real_download(filename, info_dict)           
  File "/home/user/.local/lib/python3.6/site-packages/youtube_dl/downloader/hls.py", line 163, in real_download
    'giving up after %s fragment retries' % fragment_retries)
  File "/home/user/.local/lib/python3.6/site-packages/youtube_dl/downloader/common.py", line 165, in report_error
    self.ydl.report_error(*args, **kargs)                    
  File "/home/user/.local/lib/python3.6/site-packages/youtube_dl/YoutubeDL.py", line 620, in report_error
    self.trouble(error_message, tb)                          
  File "/home/user/.local/lib/python3.6/site-packages/youtube_dl/YoutubeDL.py", line 582, in trouble
    tb_data = traceback.format_list(traceback.extract_stack())

ghost commented 6 years ago

@Troyciv youtube-dl version 2018.06.11 with --cookies option returns:

ERROR: Globo returned error: Video is for subscribers only. Authorization required.

i'm using FVD and still works, so this is a youtube-dl problem only.

Troyciv commented 6 years ago

i think yours is a different problem. does -u username -p password give you the same error?

ghost commented 6 years ago

@Troyciv you are right! this is a cookie problem only because youtube-dl -f hls-515-1 -u username -p password https://globoplay.globo.com/v/6812462/programa/ works like a charm while with valid --cookies returns this:

[Globo] 6812462: Downloading JSON metadata
[Globo] 6812462: Downloading security hash for 6812462_90_720_collection
ERROR: Globo returned error: Video is for subscribers only. Authorization required.

Troyciv commented 6 years ago

I had a cookie problem recently, turned out it was because I had automatic cookie deletion activated in my browser

Read here

ghost commented 6 years ago

@Troyciv Bingo! The cookie expiration block the download, thank you! Problem solved =D

ghost commented 6 years ago

@Troyciv The 20 minutes problem may be associated with some Globo Play mechanism for protection against ad blockers.

Troyciv commented 6 years ago

I also suspect that it is some weird server behavior, because of the constant 20min

I'm working at a python wrapper script right now that re-runs youtube-dl until no more error gets raised.

But because I'm really not a coder that might take a little while.

This of course is not a fix but it's the only thing that I am capable to do by myself.

Troyciv commented 6 years ago

Here is a wrapper script that runs youtube-dl until it terminates without any error (code 0)

usage: yt-wrapper.py -u <user> -p <password> -f <format> URL

use yt-wrapper.py -h to see help

ytdl-org / youtube-dl