search

ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
131.37k stars 9.96k forks source link

France TV 2 journal not downloading any more #29969

Closed kdliss closed 2 years ago

kdliss commented 3 years ago

since about yesterday, the daily news journal does not download any more, from FRance TV2: https://www.france.tv/france-2/journal-20h00/2743433-edition-du-mardi-14-septembre-2021.html

moviuro commented 3 years ago

Also broken, the live video feed:

% mpv --ytdl-raw-options-add=verbose='' 'https://www.france.tv/franceinfo/direct.html'
[ytdl_hook] [debug] System config: []
[ytdl_hook] [debug] User config: ['-o', '%(uploader)s-%(upload_date)s-%(title)s.%(ext)s']
[ytdl_hook] [debug] Custom config: []
[ytdl_hook] [debug] Command-line args: ['--no-warnings', '-J', '--flat-playlist', '--sub-format', 'ass/srt/best', '--format', 'bestvideo+bestaudio/best', '--verbose', '--all-subs', '--no-playlist', '--', 'https://www.france.tv/franceinfo/direct.html']
[ytdl_hook] [debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[ytdl_hook] [debug] youtube-dl version 2021.06.06
[ytdl_hook] [debug] Python version 3.9.7 (CPython) - Linux-5.12.18-1-ck-skylake-x86_64-with-glibc2.33
[ytdl_hook] [debug] exe versions: ffmpeg 4.4, ffprobe 4.4
[ytdl_hook] [debug] Proxy map: {}
[ytdl_hook] ERROR: Unable to download JSON metadata: HTTP Error 404: Not Found (caused by <HTTPError 404: 'Not Found'>); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
[ytdl_hook]   File "/usr/lib/python3.9/site-packages/youtube_dl/extractor/common.py", line 634, in _request_webpage
[ytdl_hook]     return self._downloader.urlopen(url_or_request)
[ytdl_hook]   File "/usr/lib/python3.9/site-packages/youtube_dl/YoutubeDL.py", line 2288, in urlopen
[ytdl_hook]     return self._opener.open(req, timeout=self._socket_timeout)
[ytdl_hook]   File "/usr/lib/python3.9/urllib/request.py", line 523, in open
[ytdl_hook]     response = meth(req, response)
[ytdl_hook]   File "/usr/lib/python3.9/urllib/request.py", line 632, in http_response
[ytdl_hook]     response = self.parent.error(
[ytdl_hook]   File "/usr/lib/python3.9/urllib/request.py", line 561, in error
[ytdl_hook]     return self._call_chain(*args)
[ytdl_hook]   File "/usr/lib/python3.9/urllib/request.py", line 494, in _call_chain
[ytdl_hook]     result = func(*args)
[ytdl_hook]   File "/usr/lib/python3.9/urllib/request.py", line 641, in http_error_default
[ytdl_hook]     raise HTTPError(req.full_url, code, msg, hdrs, fp) 
[ytdl_hook] youtube-dl failed: unexpected error occurred 
[ytdl_hook] It appears that your youtube-dl version is severely out of date. 
Failed to recognize file format.
% youtube-dl --version
2021.06.06
lcheylus commented 3 years ago

Currently, there is an issue with all France Televisions sites : see #29956

dirkf commented 3 years ago

The videoId (b527f895-e3f7-47b4-b9a9-62614f5ab866 in this case) seems not be in the locations where the FranceTVSiteIE searches, but in the JSON assigned to the JS variablewindow.FTVPlayerVideos, pretty-printed below:

window.FTVPlayerVideos = [
  {
    'contentId': 2743433,
    'videoId': 'b527f895-e3f7-47b4-b9a9-62614f5ab866',
    'endDate': '2038-01-19T04:14:07+01:00',
    'originUrl': '/france-2/journal-20h00/2743433-edition-du-mardi-14-septembre-2021.html',
    'comingNext': [
    ],
    'isSponsored': false,
    'isAdVisible': true,
    'videoTitle': 'Édition du mardi 14 septembre 2021',
    'programId': 2383,
    'programName': 'Journal 20h00',
    'seasonNumber': null,
    'episodeNumber': null,
    'layerType': 'default',
    'ratingCsaCode': 'TP',
    'logo': null,
    'name': null,
    'broadcastBeginDate': {
      'date': '2021-09-14 19:58:02.000000',
      'timezone_type': 1,
      'timezone': '+02:00'
    },
    'intro': true
  }
];

This may need to be added to the patch in #29956.

Trit34 commented 3 years ago

Also broken, the live video feed:

With the quick patch from #29956, you can have them working again:

[FranceTVSite] 2743433-edition-du-mardi-14-septembre-2021: Downloading webpage
[FranceTV] b527f895-e3f7-47b4-b9a9-62614f5ab866: Downloading video JSON
[FranceTV] b527f895-e3f7-47b4-b9a9-62614f5ab866: Downloading signed dash manifest URL
[FranceTV] b527f895-e3f7-47b4-b9a9-62614f5ab866: Downloading MPD manifest
WARNING: [FranceTV] Unknown MIME type application/mp4 in DASH manifest
[info] Available formats for b527f895-e3f7-47b4-b9a9-62614f5ab866:
format code           extension  resolution note
dash-audio_fre=96000  m4a        audio only [fr] DASH audio   96k , m4a_dash container, mp4a.40.2 (48000Hz)
dash-video=400000     mp4        384x216    DASH video  400k , mp4_dash container, avc1.42C01E, 25fps, video only
dash-video=950000     mp4        640x360    DASH video  950k , mp4_dash container, avc1.4D401F, 25fps, video only
dash-video=1400000    mp4        960x540    DASH video 1400k , mp4_dash container, avc1.4D401F, 25fps, video only
dash-video=2000000    mp4        1280x720   DASH video 2000k , mp4_dash container, avc1.64001F, 25fps, video only (best)

$ youtube-dl -F https://www.france.tv/france-2/direct.html
[FranceTVSite] direct: Downloading webpage
[FranceTV] 006194ea-117d-4bcf-94a9-153d999c59ae: Downloading video JSON
[FranceTV] 006194ea-117d-4bcf-94a9-153d999c59ae: Downloading signed hls manifest URL
[FranceTV] 006194ea-117d-4bcf-94a9-153d999c59ae: Downloading m3u8 information
[info] Available formats for 006194ea-117d-4bcf-94a9-153d999c59ae:
format code                          extension  resolution note
hls-audio-AACL-98-Audio_Description  mp4        audio only [qad] 
hls-audio-AACL-98-Francais           mp4        audio only [fr] 
hls-audio-AACL-98-Version_Originale  mp4        audio only [qaa] 
hls-234                              mp4        256x144     234k , avc1.42c01e, 25.0fps, video only
hls-533                              mp4        384x216     533k , avc1.42c01e, 25.0fps, video only
hls-1116                             mp4        640x360    1116k , avc1.42c01e, 25.0fps, video only
hls-1593                             mp4        960x540    1593k , avc1.4d401f, 25.0fps, video only
hls-2865                             mp4        1280x720   2865k , avc1.4d401f, 25.0fps, video only (best)

Don’t forget to add both video and audio formats to avoid getting mute videos.

lcheylus commented 3 years ago

The videoId (b527f895-e3f7-47b4-b9a9-62614f5ab866 in this case) seems not be in the locations where the FranceTVSiteIE searches, but in the JSON assigned to the JS variablewindow.FTVPlayerVideos, pretty-printed below:

No, don't need to parse JS variablewindow.FTVPlayerVideos with FranceTVSite extractor/regexp to find videoid.

With patch from #29956, youtube-dl finds correctly videoid and gets audio/video formats for your URL https://www.france.tv/france-2/journal-20h00/2743433-edition-du-mardi-14-septembre-2021.html

dirkf commented 3 years ago

The videoId (b527f895-e3f7-47b4-b9a9-62614f5ab866 in this case) seems not be in the locations where the FranceTVSiteIE searches, but in the JSON assigned to the JS variablewindow.FTVPlayerVideos, pretty-printed below:

No, don't need to parse JS variablewindow.FTVPlayerVideos with FranceTVSite extractor/regexp to find videoid.

With patch from #29956, youtube-dl finds correctly videoid and gets audio/video formats for your URL https://www.france.tv/france-2/journal-20h00/2743433-edition-du-mardi-14-septembre-2021.html

So it does. There's an alternative search pattern that I missed, so the first search attempt catches the videoId either as the value of a data-main-video attribute in an HTML tag, or as the value of a videoId JSON property, as in the above JS fragment.

However parsing the JSON fully would provide better metadata, such as these values that are null now:

Also it would be good to set duration, and, as discussed in the patch thread, subtitles.