search

ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
131.46k stars 9.96k forks source link

Using external downloader (aria2c), --min-filesize flag gets ignored #26370

Open bash-hike opened 4 years ago

bash-hike commented 4 years ago

Checklist

Verbose log

youtube-dl -v WR1r_85bOZU --ignore-config --min-filesize 1000M
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'WR1r_85bOZU', '--ignore-config', '--min-filesize', '1000M']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2020.07.28
[debug] Python version 3.8.2 (CPython) - Linux-5.4.0-42-generic-x86_64-with-glibc2.29
[debug] exe versions: ffmpeg 4.2.4, ffprobe 4.2.4, rtmpdump 2.4
[debug] Proxy map: {}
[youtube] WR1r_85bOZU: Downloading webpage
[debug] Default format spec: bestvideo+bestaudio/best
WARNING: Requested formats are incompatible for merge and will be merged into mkv.
[debug] Invoking downloader on 'https://r1---sn-ci5gup-w5pl.googlevideo.com/videoplayback?expire=1597802960&ei=cDU8X93ZCfLx4-EP2vml4AU&ip=106.207.183.178&id=o-AACWxC4nlIQrosc0oQ9RNXRR52ySBGNgd9RjOqMHays5&itag=248&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278&source=youtube&requiressl=yes&mh=mS&mm=31%2C29&mn=sn-ci5gup-w5pl%2Csn-ci5gup-qxad&ms=au%2Crdu&mv=m&mvi=1&pl=22&initcwndbps=300000&vprv=1&mime=video%2Fwebm&gir=yes&clen=77815639&dur=562.800&lmt=1558865123217304&mt=1597781236&fvip=11&keepalive=yes&fexp=23883098&c=WEB&txp=5535432&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIhAOBQ_pSORcqZyYRBUCkwXQ816P_meCkuOQlaP8W6l2qcAiBVQsexoKMm-FQ6DIqLfJsSRiHr0t5KiHa98YD-tI0vsw%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRAIgM6voSKZ7vhvDHCX9GkNNHAu8kqcPhXfNaf_9p-LOxIwCICmpLJaT5X92YXRC0VX1YEhtBiuYSjLYE7w38r_hFOHZ&ratebypass=yes'
[download] File is smaller than min-filesize (10471683 bytes < 1048576000 bytes). Aborting.
[debug] Invoking downloader on 'https://r1---sn-ci5gup-w5pl.googlevideo.com/videoplayback?expire=1597802960&ei=cDU8X93ZCfLx4-EP2vml4AU&ip=106.207.183.178&id=o-AACWxC4nlIQrosc0oQ9RNXRR52ySBGNgd9RjOqMHays5&itag=140&source=youtube&requiressl=yes&mh=mS&mm=31%2C29&mn=sn-ci5gup-w5pl%2Csn-ci5gup-qxad&ms=au%2Crdu&mv=m&mvi=1&pl=22&initcwndbps=300000&vprv=1&mime=audio%2Fmp4&gir=yes&clen=9109887&dur=562.851&lmt=1558864608181142&mt=1597781236&fvip=11&keepalive=yes&fexp=23883098&c=WEB&txp=5531432&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRgIhAIEkwii4e2_r7CAfJRWq4Fvdbq7E2fGFfNW0qq3e5RjyAiEA_WubEWBVr7xWX97XM41WxqYYkUNA4-JlbD2Pqjgoksc%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRAIgM6voSKZ7vhvDHCX9GkNNHAu8kqcPhXfNaf_9p-LOxIwCICmpLJaT5X92YXRC0VX1YEhtBiuYSjLYE7w38r_hFOHZ&ratebypass=yes'
[download] File is smaller than min-filesize (9109887 bytes < 1048576000 bytes). Aborting.
youtube-dl -v WR1r_85bOZU --ignore-config --min-filesize 1000M --external-downloader aria2c
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'WR1r_85bOZU', '--ignore-config', '--min-filesize', '1000M', '--external-downloader', 'aria2c']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2020.07.28
[debug] Python version 3.8.2 (CPython) - Linux-5.4.0-42-generic-x86_64-with-glibc2.29
[debug] exe versions: ffmpeg 4.2.4, ffprobe 4.2.4, rtmpdump 2.4
[debug] Proxy map: {}
[youtube] WR1r_85bOZU: Downloading webpage
[debug] Default format spec: bestvideo+bestaudio/best
WARNING: Requested formats are incompatible for merge and will be merged into mkv.
[debug] Invoking downloader on 'https://r1---sn-ci5gup-w5pl.googlevideo.com/videoplayback?expire=1597803024&ei=sDU8X6__Bu3_4-EPv9uByA8&ip=106.207.183.178&id=o-AES1XdJQzi4zHYzFFbgB_b7-NrC_MHKgkE5lBkHD1dcI&itag=248&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278&source=youtube&requiressl=yes&mh=mS&mm=31%2C29&mn=sn-ci5gup-w5pl%2Csn-ci5gup-qxad&ms=au%2Crdu&mv=m&mvi=1&pl=22&initcwndbps=300000&vprv=1&mime=video%2Fwebm&gir=yes&clen=77815639&dur=562.800&lmt=1558865123217304&mt=1597781300&fvip=11&keepalive=yes&fexp=23883098&c=WEB&txp=5535432&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIhALPxeh0_kvIaPbgTzSfSdZboIBg1KtSpDejb5OiohmCSAiAJsELVBo7M4UKP0z4I_s3O53CQ59oW3vSAWd0AoLZvuA%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAONrQ_4AkXdWIIq8avfjfmEt1vYMVewQPBEW7thJYUN8AiEAgaBUGOomQ0s8YsRDpCacyq1pAUVh31hUvQM8Q276Fro%3D&ratebypass=yes'
[download] Destination: The Very Best of Matt Berry as Douglas Reynholm! _ The IT Crowd-WR1r_85bOZU.f248.webm
[debug] aria2c command line: aria2c -c --min-split-size 1M --max-connection-per-server 4 --out 'The Very Best of Matt Berry as Douglas Reynholm! _ The IT Crowd-WR1r_85bOZU.f248.webm.part' --header 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3596.2 Safari/537.36' --header 'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7' --header 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' --header 'Accept-Encoding: gzip, deflate' --header 'Accept-Language: en-us,en;q=0.5' --check-certificate=true --remote-time=true -- 'https://r1---sn-ci5gup-w5pl.googlevideo.com/videoplayback?expire=1597803024&ei=sDU8X6__Bu3_4-EPv9uByA8&ip=106.207.183.178&id=o-AES1XdJQzi4zHYzFFbgB_b7-NrC_MHKgkE5lBkHD1dcI&itag=248&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C278&source=youtube&requiressl=yes&mh=mS&mm=31%2C29&mn=sn-ci5gup-w5pl%2Csn-ci5gup-qxad&ms=au%2Crdu&mv=m&mvi=1&pl=22&initcwndbps=300000&vprv=1&mime=video%2Fwebm&gir=yes&clen=77815639&dur=562.800&lmt=1558865123217304&mt=1597781300&fvip=11&keepalive=yes&fexp=23883098&c=WEB&txp=5535432&sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIhALPxeh0_kvIaPbgTzSfSdZboIBg1KtSpDejb5OiohmCSAiAJsELVBo7M4UKP0z4I_s3O53CQ59oW3vSAWd0AoLZvuA%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAONrQ_4AkXdWIIq8avfjfmEt1vYMVewQPBEW7thJYUN8AiEAgaBUGOomQ0s8YsRDpCacyq1pAUVh31hUvQM8Q276Fro%3D&ratebypass=yes'

08/19 01:40:25 [NOTICE] Downloading 1 item(s)
[#961995 74MiB/74MiB(99%) CN:1 DL:1.3MiB]                                                                                                                                                    
08/19 01:41:28 [NOTICE] Download complete: /home/krishna/The Very Best of Matt Berry as Douglas Reynholm! _ The IT Crowd-WR1r_85bOZU.f248.webm.part

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
961995|OK  |   1.1MiB/s|/home/krishna/The Very Best of Matt Berry as Douglas Reynholm! _ The IT Crowd-WR1r_85bOZU.f248.webm.part

Status Legend:
(OK):download completed.
[aria2c] Downloaded 77815639 bytes
[download] 100% of 74.21MiB in 01:03
[debug] Invoking downloader on 'https://r1---sn-ci5gup-w5pl.googlevideo.com/videoplayback?expire=1597803024&ei=sDU8X6__Bu3_4-EPv9uByA8&ip=106.207.183.178&id=o-AES1XdJQzi4zHYzFFbgB_b7-NrC_MHKgkE5lBkHD1dcI&itag=140&source=youtube&requiressl=yes&mh=mS&mm=31%2C29&mn=sn-ci5gup-w5pl%2Csn-ci5gup-qxad&ms=au%2Crdu&mv=m&mvi=1&pl=22&initcwndbps=300000&vprv=1&mime=audio%2Fmp4&gir=yes&clen=9109887&dur=562.851&lmt=1558864608181142&mt=1597781300&fvip=11&keepalive=yes&fexp=23883098&c=WEB&txp=5531432&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIhAM6E8lbzR16vfECiHCaaa8Cdu58zwuo8w5wrbC_1T3XuAiArj70zGieIEd7rSVt_uQlqGrWB65wDoIFInMj1Oiz_dA%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAONrQ_4AkXdWIIq8avfjfmEt1vYMVewQPBEW7thJYUN8AiEAgaBUGOomQ0s8YsRDpCacyq1pAUVh31hUvQM8Q276Fro%3D&ratebypass=yes'
[download] Destination: The Very Best of Matt Berry as Douglas Reynholm! _ The IT Crowd-WR1r_85bOZU.f140.m4a
[debug] aria2c command line: aria2c -c --min-split-size 1M --max-connection-per-server 4 --out 'The Very Best of Matt Berry as Douglas Reynholm! _ The IT Crowd-WR1r_85bOZU.f140.m4a.part' --header 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3596.2 Safari/537.36' --header 'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7' --header 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' --header 'Accept-Encoding: gzip, deflate' --header 'Accept-Language: en-us,en;q=0.5' --check-certificate=true --remote-time=true -- 'https://r1---sn-ci5gup-w5pl.googlevideo.com/videoplayback?expire=1597803024&ei=sDU8X6__Bu3_4-EPv9uByA8&ip=106.207.183.178&id=o-AES1XdJQzi4zHYzFFbgB_b7-NrC_MHKgkE5lBkHD1dcI&itag=140&source=youtube&requiressl=yes&mh=mS&mm=31%2C29&mn=sn-ci5gup-w5pl%2Csn-ci5gup-qxad&ms=au%2Crdu&mv=m&mvi=1&pl=22&initcwndbps=300000&vprv=1&mime=audio%2Fmp4&gir=yes&clen=9109887&dur=562.851&lmt=1558864608181142&mt=1597781300&fvip=11&keepalive=yes&fexp=23883098&c=WEB&txp=5531432&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIhAM6E8lbzR16vfECiHCaaa8Cdu58zwuo8w5wrbC_1T3XuAiArj70zGieIEd7rSVt_uQlqGrWB65wDoIFInMj1Oiz_dA%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAONrQ_4AkXdWIIq8avfjfmEt1vYMVewQPBEW7thJYUN8AiEAgaBUGOomQ0s8YsRDpCacyq1pAUVh31hUvQM8Q276Fro%3D&ratebypass=yes'

08/19 01:41:28 [NOTICE] Downloading 1 item(s)
[#e3da8f 8.1MiB/8.6MiB(94%) CN:1 DL:1.1MiB]                                                                                                                                                  
08/19 01:41:36 [NOTICE] Download complete: /home/krishna/The Very Best of Matt Berry as Douglas Reynholm! _ The IT Crowd-WR1r_85bOZU.f140.m4a.part

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
e3da8f|OK  |   1.1MiB/s|/home/krishna/The Very Best of Matt Berry as Douglas Reynholm! _ The IT Crowd-WR1r_85bOZU.f140.m4a.part

Status Legend:
(OK):download completed.
[aria2c] Downloaded 9109887 bytes
[download] 100% of 8.69MiB in 00:08
[ffmpeg] Merging formats into "The Very Best of Matt Berry as Douglas Reynholm! _ The IT Crowd-WR1r_85bOZU.mkv"
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:The Very Best of Matt Berry as Douglas Reynholm! _ The IT Crowd-WR1r_85bOZU.f248.webm' -i 'file:The Very Best of Matt Berry as Douglas Reynholm! _ The IT Crowd-WR1r_85bOZU.f140.m4a' -c copy -map 0:v:0 -map 1:a:0 'file:The Very Best of Matt Berry as Douglas Reynholm! _ The IT Crowd-WR1r_85bOZU.temp.mkv'
Deleting original file The Very Best of Matt Berry as Douglas Reynholm! _ The IT Crowd-WR1r_85bOZU.f248.webm (pass -k to keep)
Deleting original file The Very Best of Matt Berry as Douglas Reynholm! _ The IT Crowd-WR1r_85bOZU.f140.m4a (pass -k to keep)

Description

The --min-filesize flag gets ignored if --external-downloader is given too. This is true for at least aria2 as the external downloader, and has been confirmed on multiple systems. The order in which arguments are passed does not matter (as it should be). And it is just the filesize selection argument that gets ignored, passing something like --min-views 10000000000 skips the download.

Another bug that is probably related is getting thumbnails.

--min-views 100000000 --write-thumbnail skips the video, and hence, doesn't download the thumbnail. However,

--min-filesize 1000M --write-thumbnail skips the video, but still downloads the thumbnail.

blackjack4494 commented 4 years ago

You can pass arguments to external downloader with --external-downloader-args. However aria2c doesn't have some min or max file feature. --min-filesize won't work when used with external downloader as youtube-dl won't have a clue how big the file is. There could be a workaround by checking the file(s) metadata (ContentRange) but that only works if this metadata is returned by the server if you probe the download. Also if there is a playlist file like m3u8 or so there will be quite some smaller files anyway. Another workaround would be to delete the file after it's downloaded and is below threshold. But this is most likely not what you want and could be better implemented in some script outside of youtube-dl.

michealespinola commented 4 years ago

Putting it that way certainly makes some sense of it, but perhaps there should be a warning logged when using --min-filesize in conjuncture with --external-downloader-args, as many users will not grasp these technicalities.