[ARD] HD format as mp4 file (best) not found

andreygursky commented 5 years ago

Make sure you are using the latest version: run `youtube-dl --version` and ensure your version is 2018.09.26. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

[x] I've verified and I assure that I'm running youtube-dl 2018.09.26

Before submitting an issue make sure you have:

[x] At least skimmed through the README, most notably the FAQ and BUGS sections
[x] Searched the bugtracker for similar issues including closed ones
[x] Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser

What is the purpose of your issue?

[x] Bug report (encountered problems with youtube-dl)
[ ] Site support request (request for adding support for a new site)
[ ] Feature request (request for a new functionality)
[ ] Question
[ ] Other

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

youtube-dl -v -F 'https://www.ardmediathek.de/tv/Babylon-Berlin/1929-Das-Jahr-Babylon/Das-Erste/Video?bcastId=54319834&documentId=56517232'
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'-F', u'https://www.ardmediathek.de/tv/Babylon-Berlin/1929-Das-Jahr-Babylon/Das-Erste/Video?bcastId=54319834&documentId=56517232']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.09.26
[debug] Python version 2.7.15 (CPython) - Linux-4.14.0-3-amd64-x86_64-with-debian-9.0                                                                                
[debug] exe versions: ffmpeg 3.4.2-1, ffprobe 3.4.2-1, rtmpdump 2.4
[debug] Proxy map: {}
[ARD:mediathek] 56517232: Downloading webpage
[ARD:mediathek] 56517232: Downloading media JSON
[ARD:mediathek] 56517232: Downloading f4m manifest
[ARD:mediathek] 56517232: Downloading m3u8 information
[ARD:mediathek] 56517232: Downloading m3u8 information
[info] Available formats for 56517232:
format code  extension  resolution note
a0-mp4-2     mp4        unknown
a1-mp4-0     mp4        unknown
a1-mp4-1-0   mp4        unknown
a1-mp4-1-1   mp4        unknown
a1-mp4-2-0   mp4        unknown
a1-mp4-2-1   mp4        unknown
a1-mp4-3     mp4        unknown
hls-188-0    mp4        320x180     188k , avc1.66.30, mp4a.40.2
hls-188-1    mp4        320x180     188k , avc1.66.30, mp4a.40.2
hls-316-0    mp4        480x270     316k , avc1.66.30, mp4a.40.2
hls-316-1    mp4        480x270     316k , avc1.66.30, mp4a.40.2
hds-604      flv        unknown     604k
hls-604-0    mp4        512x288     604k , avc1.77.30, mp4a.40.2
hls-604-1    mp4        512x288     604k , avc1.77.30, mp4a.40.2
hds-1212     flv        unknown    1212k
hls-1212-0   mp4        640x360    1212k , avc1.77.30, mp4a.40.2
hls-1212-1   mp4        640x360    1212k , avc1.77.30, mp4a.40.2
hds-1988     flv        unknown    1988k
hls-1988-0   mp4        960x540    1988k , avc1.77.30, mp4a.40.2
hls-1988-1   mp4        960x540    1988k , avc1.77.30, mp4a.40.2
hls-3771     mp4        1280x720   3771k , avc1.64001f, mp4a.40.2 (best)

Description of your issue, suggested solution and other information

1280x720 mp4 file (https://pdvideosdaserste-a.akamaihd.net/int/2018/09/18/196f8dd1-4491-4405-be65-fff78fcc8e0d/1280-1.mp4) has not been discovered. It should be selected as "best" instead of hls-3771 ts stream (which would need ffmpeg postprocessing).

P.S. above is only an example for the issue with ARD Mediathek.

Hrxn commented 5 years ago

Well, seems like the same result, basically.. (720p) Did you check the differences between format hls-3771 and the other format you mentioned?

andreygursky commented 5 years ago

diff ffprobe HLS stream <-> HTTP file download:

--- "1929 - Das Jahr Babylon-56517232.mp4.ffprobe"  2018-10-02 14:26:43.750085877 +0200
+++ 1280-1.mp4.ffprobe  2018-10-02 14:27:02.566086703 +0200
@@ -1,13 +1,16 @@
-Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '1929 - Das Jahr Babylon-56517232.mp4':
+Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '1280-1.mp4':                                               
   Metadata:
-    major_brand     : isom
-    minor_version   : 512
-    compatible_brands: isomiso2avc1mp41
-    encoder         : Lavf57.83.100
-  Duration: 00:43:50.08, start: 0.000000, bitrate: 3777 kb/s
-    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 1280x720 [SAR 1:1 DAR 16:9], 3583 kb/s, 25 fps, 25 tbr, 90k tbn, 50 tbc (default)  
+    major_brand     : mp42
+    minor_version   : 0
+    compatible_brands: isom
+    creation_time   : 2018-09-18T20:48:26.000000Z
+  Duration: 00:43:50.08, start: 0.000000, bitrate: 3775 kb/s
+    Stream #0:0(eng): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 1280x720 [SAR 1:1 DAR 16:9], 3582 kb/s, 25 fps, 25 tbr, 25k tbn, 50 tbc (default)  
     Metadata:
-      handler_name    : VideoHandler
-    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 189 kb/s (default)                                                                
+      creation_time   : 2018-09-18T20:48:26.000000Z
+      handler_name    : MP4 Video Media Handler
+      encoder         : AVC Coding
+    Stream #0:1(deu): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 189 kb/s (default)                                                                
     Metadata:
-      handler_name    : SoundHandler
+      creation_time   : 2018-09-18T20:48:26.000000Z
+      handler_name    : MP4 Sound Media Handler

diff mediainfo HLS stream <-> HTTP file download:

--- "1929 - Das Jahr Babylon-56517232.mp4.mediainfo"    2018-10-02 14:23:12.784419862 +0200
+++ 1280-1.mp4.mediainfo        2018-10-02 14:22:49.024420168 +0200
@@ -1,13 +1,14 @@
 General
-Complete name                            : 1929 - Das Jahr Babylon-56517232.mp4
+Complete name                            : 1280-1.mp4
 Format                                   : MPEG-4
-Format profile                           : Base Media
-Codec ID                                 : isom (isom/iso2/avc1/mp41)
+Format profile                           : Base Media / Version 2
+Codec ID                                 : mp42 (isom)
 File size                                : 1.16 GiB
 Duration                                 : 43 min 50 s
-Overall bit rate mode                    : Constant
-Overall bit rate                         : 3 777 kb/s
-Writing application                      : Lavf57.83.100
+Overall bit rate mode                    : Variable
+Overall bit rate                         : 3 775 kb/s
+Encoded date                             : UTC 2018-09-18 20:48:26
+Tagged date                              : UTC 2018-09-18 20:48:26

 Video
 ID                                       : 1
@@ -35,6 +36,9 @@ Bit depth
 Scan type                                : Progressive
 Bits/(Pixel*Frame)                       : 0.156
 Stream size                              : 1.10 GiB (95%)
+Language                                 : English
+Encoded date                             : UTC 2018-09-18 20:48:26
+Tagged date                              : UTC 2018-09-18 20:48:26
 Color range                              : Limited
 Color primaries                          : BT.709
 Transfer characteristics                 : BT.709
@@ -47,15 +51,17 @@ Format/Info
 Format profile                           : LC
 Codec ID                                 : mp4a-40-2
 Duration                                 : 43 min 50 s
-Bit rate mode                            : Constant
+Bit rate mode                            : Variable
 Bit rate                                 : 192 kb/s
+Maximum bit rate                         : 305 kb/s
 Channel(s)                               : 2 channels
 Channel positions                        : Front: L R
 Sampling rate                            : 48.0 kHz
 Frame rate                               : 46.875 FPS (1024 SPF)
 Compression mode                         : Lossy
 Stream size                              : 59.4 MiB (5%)
-Default                                  : Yes
-Alternate group                          : 1
+Language                                 : German
+Encoded date                             : UTC 2018-09-18 20:48:26
+Tagged date                              : UTC 2018-09-18 20:48:26

HLS stream:

need to call ffmpeg
download time: 05:17

HTTP file download:

faster download: 03:33
encoding timestamp preserved
audio language preserved
md5 checksum provided

Hrxn commented 5 years ago

And bitrate 3583 kb/s vs. 3582 kb/s.. 😅

No really, thanks for this comparison. And I agree, the MP4 direct download is actually the better default and should be preferred, for the reasons you already mentioned.

BTW, what did you use to find the links? One of these?

https://mediathekdirekt.de/
https://mediathekviewweb.de/

andreygursky commented 5 years ago

BTW, what did you use to find the links? One of these?

Exactly. https://mediathekviewweb.de - is my favorite because of being fast due to using AJAX. In case I need a database in a local file, I go to https://mediathekdirekt.de/.

ngdio commented 5 years ago

Could you try out the version from my pull request? It solve this issue for all media from Das Erste.

andreygursky commented 4 years ago

Sorry for not replying before. For now the patch cannot be cleanly applied. But looking at https://github.com/ytdl-org/youtube-dl/pull/17851/files#diff-2ebe82369b35395b419db1575a615d9dR27

    # IMPORTANT: Not valid for media from WDR/BR/etc. servers

it seems it is not exactly what this issue is about, since the video from 'Das Erste' was only one example for an ARD URL (www.ardmediathek.de / classic.ardmediathek.de).

andreygursky commented 4 years ago

For ARD download URLs can be obtained generally as follows (in a bash shell):

URL_CLASSIC='https://classic.ardmediathek.de/tv/mareTV/Estlands-verwunschene-Inseln/NDR-Fernsehen/Video?bcastId=14049152&documentId=66593126'
ID=`echo "${URL_CLASSIC}" | sed -e 's|.*documentId=\([0-9]*\)|\1|'`
JSON=`wget -q -O - "https://www.ardmediathek.de/play/media/${ID}"`
echo $JSON | jq -r '._mediaArray[]._mediaStreamArray[]._stream'

Result:

https://ndrod-vh.akamaihd.net/i/ndr/2019/0909/TV-20190909-1144-2800.,hd,hq,ln,hi,mn,lo,ao,.mp4.csmil/master.m3u8
https://mediandr-a.akamaihd.net/progressive/2019/0909/TV-20190909-1144-2800.lo.mp4
[
  "https://mediandr-a.akamaihd.net/progressive/2019/0909/TV-20190909-1144-2800.ln.mp4",
  "https://mediandr-a.akamaihd.net/progressive/2019/0909/TV-20190909-1144-2800.hi.mp4",
  "https://mediandr-a.akamaihd.net/progressive/2019/0909/TV-20190909-1144-2800.mn.mp4"
]
https://mediandr-a.akamaihd.net/progressive/2019/0909/TV-20190909-1144-2800.hq.mp4
https://mediandr-a.akamaihd.net/progressive/2019/0909/TV-20190909-1144-2800.hd.mp4

andreygursky commented 4 years ago

It looks like for quality parameter applies:

0 is "Low Quality"
1 is "Medium Quality"
2 is "High Quality" (actually "SD")
3 is "HD"
4 is "FullHD"

Thus one can select the desired download URL (e.g. "HD"):

echo "$JSON" | jq -r '._mediaArray[]._mediaStreamArray[] as $data | select($data._quality == 3) | $data._stream'

Result:

https://mediandr-a.akamaihd.net/progressive/2019/0909/TV-20190909-1144-2800.hd.mp4

ngdio commented 4 years ago

I might take a look at this again next week. Things have changed significantly since 2018, so it's likely all going to be different now. Shouldn't be difficult to find out, though.

it seems it is not exactly what this issue is about, since the video from 'Das Erste' was only one example for an ARD URL (www.ardmediathek.de / classic.ardmediathek.de).

At the time, there weren't any higher quality streams/files from WDR/BR/etc. that weren't found by the existing code. Might have changed now but only ARD-Mediathek-hosted videos had higher quality back in 2018.

ytdl-org / youtube-dl