External downloader argument ignored - fragment downloader used for mp4 stream

[DoctorVanGogh]

Please follow the guide below

• You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
• Put an x into all the boxes [ ] relevant to your issue (like that [x])
• Use Preview tab to see how your issue will actually look like

---

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2016.10.12. If it's not read this FAQ entry and update. Issues with outdated version will be rejected.

• [x] I've verified and I assure that I'm running youtube-dl 2016.10.12

  Before submitting an issue make sure you have:

• [x] At least skimmed through README and most notably FAQ and BUGS sections
• [x] Searched the bugtracker for similar issues including closed ones

  What is the purpose of your issue?

• [x] Bug report (encountered problems with youtube-dl)
• [ ] Site support request (request for adding support for a new site)
• [ ] Feature request (request for a new functionality)
• [ ] Question
• [ ] Other

---

Error:

Youtube-dl seems to ignore the external-downloader argument and unnecessarily download certain streams using it's internal 'fragment' downloader.

Example

https://www.youtube.com/watch?v=ZQJwjSjZJ0I

Invoking a download with youtube-dl --ffmpeg-location c:\ffmpeg\bin --external-downloader aria2c --external-downloader-args "-j 8 -s 8 -x 8 -k 5M" https://www.youtube.com/watch?v=ZQJwjSjZJ0I should download two simple partial streams for subsequent joining by ffmpeg (I think it's trying the equivalent of -f 290+140 for the given video). Those partial stream downloads are not passed to the external downloader, but processed internally by the (m3u8?) fragment downloader. Notice the 'Total fragments: 418' line in the output:

> youtube-dl --ffmpeg-location c:\ffmpeg\bin --external-downloader aria2c --external-downloader-args "-j 8 -s 8 -x 8 -k 5M" https://www.youtube.com/watch?v=ZQJwjSjZJ0I -v
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['--ffmpeg-location', 'c:\\ffmpeg\\bin', '--external-downloader', 'aria2c', '--external-downloader-args', '-j 8 -s 8 -x 8 -k 5M', 'https://www.youtube.com/watch?v=ZQJwjSjZJ0I', '-v']
[debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252
[debug] youtube-dl version 2016.10.12
[debug] Python version 3.4.4 - Windows-10-10.0.10586
[debug] exe versions: ffmpeg N-80256-g0a9e781, ffprobe N-80256-g0a9e781
[debug] Proxy map: {}
[youtube] ZQJwjSjZJ0I: Downloading webpage
[youtube] ZQJwjSjZJ0I: Downloading video info webpage
[youtube] ZQJwjSjZJ0I: Extracting video information
[youtube] ZQJwjSjZJ0I: Downloading MPD manifest
[youtube] ZQJwjSjZJ0I: Downloading MPD manifest
[debug] Invoking downloader on 'https://r3---sn-4g5edne6.googlevideo.com/videoplayback/id/6502708d28d92742/itag/299/source/youtube/requiressl/yes/nh/IgpwZjAyLmZyYTE1Kg03Mi4xNC4yMDMuMjMx/initcwndbps/571250/pl/19/mv/m/ms/au/mm/31/mn/sn-4g5edne6/ratebypass/yes/mime/video%2Fmp4/otfp/1/gir/yes/clen/678034295/lmt/1476448414314165/dur/2113.033/signature/917DAF6693CAF110318D761FDCD13780EB9E1302.561B90DBBD631CD7F4228224D4960F4BB373B294/upn/tyycdpZoj38/key/dg_yt0/mt/1476471461/ip/*IPREMOVED*/ipbits/0/expire/1476493372/sparams/ip,ipbits,expire,id,itag,source,requiressl,nh,initcwndbps,pl,mv,ms,mm,mn,ratebypass,mime,otfp,gir,clen,lmt,dur/'
[dashsegments] Total fragments: 418
[download] Destination: Nemo Plays - Starsector A Pirate's Life #10 - Massive Profits!-ZQJwjSjZJ0I.f299.mp4
[download]   4.2% of ~405.61MiB at  1.46MiB/s ETA 05:35
ERROR: Interrupted by user

During download the target directory accumulates a lot of files in the [Filename].[Format].ext.part-FragNNN naming scheme.

While this download works, it takes unnecessarily long - especially since the external downloader would have processed things way faster.

Workaround

The following workaround get's things downloaded faster:

> youtube-dl -g --ffmpeg-location c:\ffmpeg\bin https://www.youtube.com/watch?v=ZQJwjSjZJ0I > locations -v
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['-g', '--ffmpeg-location', 'c:\\ffmpeg\\bin', 'https://www.youtube.com/watch?v=ZQJwjSjZJ0I', '-v']
[debug] Encodings: locale cp1252, fs mbcs, out cp1252, pref cp1252
[debug] youtube-dl version 2016.10.12
[debug] Python version 3.4.4 - Windows-10-10.0.10586
[debug] exe versions: ffmpeg N-80256-g0a9e781, ffprobe N-80256-g0a9e781
[debug] Proxy map: {}

>aria2c -i locations -j 8 -s 8 -x 8 -k 5M

10/14 20:13:10 [NOTICE] Downloading 2 item(s)

10/14 20:13:10 [NOTICE] Allocating disk space. Use --file-allocation=none to disable it. See --file-allocation option in man page for more details.

10/14 20:13:10 [NOTICE] File already exists. Renamed to C:/Temp/index.1.html.
[DL:10MiB][#97df67 270MiB/646MiB(41%)][#465dba 31MiB/32MiB(99%)]
10/14 20:13:41 [NOTICE] Download complete: C:/Temp/index.1.html
 *** Download Progress Summary as of Fri Oct 14 20:14:10 2016 ***
======================================================================================================================
[#97df67 557MiB/646MiB(86%) CN:8 DL:9.6MiB ETA:9s]
FILE: C:/Temp/index.html
----------------------------------------------------------------------------------------------------------------------

[#97df67 646MiB/646MiB(99%) CN:1 DL:1.6MiB]
10/14 20:14:29 [NOTICE] Download complete: C:/Temp/index.html

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
465dba|OK  |   1.0MiB/s|C:/Temp/index.1.html
97df67|OK  |   8.2MiB/s|C:/Temp/index.html

Status Legend:
(OK):download completed.

> c:\FFMPEG\bin\ffmpeg.exe -i index.1.html -i index.html -c copy result.mkv

Total ellapsed time using workaround: ~1min 30, about a third what plain youtube-dl invocation returns.

Counter-Example

Here's another stream, where youtube-dl exhibits exactly the expected behaviour and passes the actual downloads on to the external client:

youtube-dl --ffmpeg-location c:\ffmpeg\bin --external-downloader aria2c --external-downloader-args "-j 8 -s 8 -x 8 -k 5M" https://www.youtube.com/watch?v=H7Uyfqi_TE8 -v
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['--ffmpeg-location', 'c:\\ffmpeg\\bin', '--external-downloader', 'aria2c', '--external-downloader-args', '-j 8 -s 8 -x 8 -k 5M', 'https://www.youtube.com/watch?v=H7Uyfqi_TE8', '-v']
[debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252
[debug] youtube-dl version 2016.10.12
[debug] Python version 3.4.4 - Windows-10-10.0.10586
[debug] exe versions: ffmpeg N-80256-g0a9e781, ffprobe N-80256-g0a9e781
[debug] Proxy map: {}
[youtube] H7Uyfqi_TE8: Downloading webpage
[youtube] H7Uyfqi_TE8: Downloading video info webpage
[youtube] H7Uyfqi_TE8: Extracting video information
[youtube] H7Uyfqi_TE8: Downloading MPD manifest
[debug] Invoking downloader on 'https://r2---sn-4g5edned.googlevideo.com/videoplayback?id=1fb5327ea8bf4c4f&itag=266&source=youtube&requiressl=yes&pl=19&mm=31&mn=sn-4g5edned&ms=au&mv=m&initcwndbps=626250&nh=IgpwZjAxLmZyYTE2Kg04MC4xNTAuMTcwLjY5&ratebypass=yes&mime=video/mp4&gir=yes&clen=4256779734&lmt=1475157498730931&dur=3885.547&key=dg_yt0&mt=1476472455&signature=88D175636E0E1AD0D541BB5A2B15B464EC23F89C.7E4573730D43A2C2B8E644EF9E46D263C9278CA8&upn=UmsUXDZf0x0&ip=*removed*&ipbits=0&expire=1476494647&sparams=ip,ipbits,expire,id,itag,source,requiressl,pl,mm,mn,ms,mv,initcwndbps,nh,ratebypass,mime,gir,clen,lmt,dur'
[download] Destination: Making Humans a Multiplanetary Species-H7Uyfqi_TE8.f266.mp4
[debug] aria2c command line: aria2c -c -j 8 -s 8 -x 8 -k 5M --out 'Making Humans a Multiplanetary Species-H7Uyfqi_TE8.f266.mp4.part' --header 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' --header 'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7' --header 'Accept-Encoding: gzip, deflate' --header 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/47.0 (Chrome)' --header 'Accept-Language: en-us,en;q=0.5' --check-certificate=true -- 'https://r2---sn-4g5edned.googlevideo.com/videoplayback?id=1fb5327ea8bf4c4f&itag=266&source=youtube&requiressl=yes&pl=19&mm=31&mn=sn-4g5edned&ms=au&mv=m&initcwndbps=626250&nh=IgpwZjAxLmZyYTE2Kg04MC4xNTAuMTcwLjY5&ratebypass=yes&mime=video/mp4&gir=yes&clen=4256779734&lmt=1475157498730931&dur=3885.547&key=dg_yt0&mt=1476472455&signature=88D175636E0E1AD0D541BB5A2B15B464EC23F89C.7E4573730D43A2C2B8E644EF9E46D263C9278CA8&upn=UmsUXDZf0x0&ip=*removed*&ipbits=0&expire=1476494647&sparams=ip,ipbits,expire,id,itag,source,requiressl,pl,mm,mn,ms,mv,initcwndbps,nh,ratebypass,mime,gir,clen,lmt,dur'

10/14 21:24:02 [NOTICE] Downloading 1 item(s)

10/14 21:24:02 [NOTICE] Allocating disk space. Use --file-allocation=none to disable it. See --file-allocation option in man page for more details.
[#8de49c 62MiB/3.9GiB(1%) CN:8 DL:10MiB ETA:6m32s]
10/14 21:24:18 [NOTICE] Shutdown sequence commencing... Press Ctrl-C again for emergency shutdown.

10/14 21:24:18 [NOTICE] Download GID#8de49c2b6b0b517d not complete: C:/Temp/Making Humans a Multiplanetary Species-H7Uyfqi_TE8.f266.mp4.part

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
8de49c|INPR|    10MiB/s|C:/Temp/Making Humans a Multiplanetary Species-H7Uyfqi_TE8.f266.mp4.part

Status Legend:
(INPR):download in-progress.

aria2 will resume download if the transfer is restarted.
If there are any errors, then see the log file. See '-l' option in help/man page for details.

ERROR: Interrupted by user

Conclusion

I think the current youtube-dl behaviour for this example stream here is wrong. I have no idea if it's a stream or a youtube-dl issue. If an external downloader is passed then it should be used if at all possible.

I realize there are situations where this is not yet possible (m3u8 streams - see #6392), but things should work for this stream.

[dstftw]

ZQJwjSjZJ0I's bestvideo and bestaudio formats are served as dashsegments, i.e. the URLs you get with -g are not URLs to the complete downloadable videos but base URLs to the locations where the segments are stored. For some reason this particular video serves the whole files via these base URLs but that is not the case in general (e.g. aclExbIzafg). So it's legitimately ignores the external downloader and delegates to dashsegments downloader. In case of H7Uyfqi_TE8, bestvideo and bestaudio are non segmented dash and -g returns URLs to complete downloadable files thus external downloader is respected in this case.