ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
132.53k stars 10.05k forks source link

Downloading multiple subtitles of the same language from Crunchyroll #20227

Open jmaeshawn opened 5 years ago

jmaeshawn commented 5 years ago

Please follow the guide below


Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2019.03.01. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

Before submitting an issue make sure you have:

What is the purpose of your issue?


The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue


If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add the -v flag to your command line you run youtube-dl with (youtube-dl -v <your command line>), copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

H:\Install Files\Crunchyroll Ripper>"H:\Install Files\Crunchyroll Ripper\youtube-dl.exe" "https://www.crunchyroll.com/ace-attorney/episode-16-turnabout-beginnings-last-trial-782001"  --write-sub --all-subs --user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0 Waterfox/56.2.7" --cookies "H:\Install Files\Crunchyroll Ripper\cookies.txt" -v
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['https://www.crunchyroll.com/ace-attorney/episode-16-turnabout-beginnings-last-trial-782001', '--write-sub', '--all-subs', '--user-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:65.0) Gecko/20100101 Firefox/65.0 Waterfox/56.2.7', '--cookies', 'H:\\Install Files\\Crunchyroll Ripper\\cookies.txt', '-v']
[debug] Encodings: locale cp932, fs mbcs, out cp932, pref cp932
[debug] youtube-dl version 2019.03.01
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.17763
[debug] exe versions: ffmpeg 3.4.2
[debug] Proxy map: {}
[crunchyroll] 782001: Downloading webpage
[crunchyroll] 782001: Downloading audio-jaJP-hardsub-ptBR m3u8 information
[crunchyroll] 782001: Downloading audio-jaJP-hardsub-arME m3u8 information
[crunchyroll] 782001: Downloading audio-jaJP-hardsub-itIT m3u8 information
[crunchyroll] 782001: Downloading audio-jaJP-hardsub-esES m3u8 information
[crunchyroll] 782001: Downloading audio-jaJP-hardsub-frFR m3u8 information
[crunchyroll] 782001: Downloading audio-jaJP m3u8 information
[crunchyroll] 782001: Downloading audio-jaJP-hardsub-deDE m3u8 information
[crunchyroll] 782001: Downloading audio-jaJP-hardsub-enUS m3u8 information
[crunchyroll] 782001: Downloading audio-jaJP-hardsub-esLA m3u8 information
[crunchyroll] 782001: Downloading audio-jaJP-hardsub-ruRU m3u8 information
[crunchyroll] 782001: Downloading media info
[debug] Default format spec: bestvideo+bestaudio/best
[info] Writing video subtitles to: Ace Attorney Season 2 Episode 16 – Turnabout Beginnings — Last Trial-782001.ruRU.ass
[info] Writing video subtitles to: Ace Attorney Season 2 Episode 16 – Turnabout Beginnings — Last Trial-782001.esLA.ass
[info] Writing video subtitles to: Ace Attorney Season 2 Episode 16 – Turnabout Beginnings — Last Trial-782001.enUS.ass
[info] Writing video subtitles to: Ace Attorney Season 2 Episode 16 – Turnabout Beginnings — Last Trial-782001.esES.ass
[info] Writing video subtitles to: Ace Attorney Season 2 Episode 16 – Turnabout Beginnings — Last Trial-782001.frFR.ass
[info] Writing video subtitles to: Ace Attorney Season 2 Episode 16 – Turnabout Beginnings — Last Trial-782001.ptBR.ass
[info] Writing video subtitles to: Ace Attorney Season 2 Episode 16 – Turnabout Beginnings — Last Trial-782001.itIT.ass
[info] Writing video subtitles to: Ace Attorney Season 2 Episode 16 – Turnabout Beginnings — Last Trial-782001.arME.ass
[info] Writing video subtitles to: Ace Attorney Season 2 Episode 16 – Turnabout Beginnings — Last Trial-782001.deDE.ass
[debug] Invoking downloader on 'https://v.vrv.co/evs/64e835a30f11c6f1975d61e1342b2170/assets/951b32abd76f17e7ff0aae2d194f8181_3626624.mp4/index-v1-a1.m3u8?Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly92LnZydi5jby9ldnMvNjRlODM1YTMwZjExYzZmMTk3NWQ2MWUxMzQyYjIxNzAvYXNzZXRzLyoiLCJDb25kaXRpb24iOnsiRGF0ZUxlc3NUaGFuIjp7IkFXUzpFcG9jaFRpbWUiOjE1NTE3MTc4ODB9fX1dfQ__&Signature=hIrx5MPXs18f0yqGxEl9kHLfYmwLLFqOY-mFsZGh3i-xULc0r~6IrsaFWE8lKN6pserF-ZLAE69Hm6uAxE2I1m2BZeCNcRXEVokSAbCTvWYvliJ0t1yw1Mf3Y5Ym9DsoF5ZaREDDema3w2nmL88huTwbHeTJDYunU9pKbnu8Q5rNtoxhQh1j-mcVMijqjvVteycd3KPLDxMTUBlkVgC-luUOxmDIVVuJZ3gtW4~-dN0bopJj768S~vtRGMhGkmZ3m-ysOJcmvm55kcr3gdqRvCNiOrQHq6whbiLgHTJv3K~pupmy-RK4FzcumeV3RE4p-Kq2nRkJNknfS~1EZO5IIA__&Key-Pair-Id=APKAJKQQ2INNHTYFB44A'
[download] Ace Attorney Season 2 Episode 16 – Turnabout Beginnings — Last Trial-782001.mp4 has already been downloaded
[download] 100% of 800.76MiB

Description of your issue, suggested solution and other information

As you see from the above log, the issue I'm reporting doesn't have anything to do with downloading the video from Crunchyroll, since that works fine for me. My issue is that certain series (such as Ace Attorney in my above log) have two separate subtitle tracks that are both enUS, but differ in that one has the character names that were used in the English localization of the game the series is based on, and the second enUS track uses romanizations of the character names in Japanese. However, youtube-dl only seems to recognize one subtitle track per language, even when --allsubs is specified as a parameter. It also seems to decide at random which of the two enUS tracks it will download, because sometimes it will download only the track with English localized names, and other times it will download only the track with Japanese names.

Is it possible to make youtube-dl be able to recognize and download multiple subtitle tracks of the same language from Crunchyroll? For example, they could be output as "Ace Attorney Season 2 Episode 16 – Turnabout Beginnings — Last Trial-782001.enUS.ass" for the default track with localized names, and "Ace Attorney Season 2 Episode 16 – Turnabout Beginnings — Last Trial-782001.enUS.alternate.ass for the alternate track with Japanese names.

Kojirosan commented 5 years ago

At least you can download. Me, I always have this error 503 whatever I do.

darkhelmet2016 commented 5 years ago

Each one uses a different session id. But for some reason even on their website it will not let you chose the other track. Might be a bug on their end. Submitted a site request to fix.

https://www.crunchyroll.com/ace-attorney/episode-16-turnabout-beginnings-last-trial-782001?ssid=344973 https://www.crunchyroll.com/ace-attorney/episode-16-turnabout-beginnings-last-trial-782001?ssid=344996

wiznaibus commented 5 years ago

I'm having the same issue with V LIVE. They allow users to create fansubs for videos and this can result in having multiple subtitle tracks for the same language. Notably, the unofficial en_US subtitle for this video is actually in Turkish: https://www.vlive.tv/video/3697?channelCode=FE619 (When downloading the video with youtube-dl, the unofficial/Turkish en_US track is chosen over the official en_US one.)

The video shows 21 different available subtitle tracks, but when using youtube-dl --list-subs https://www.vlive.tv/video/3697?channelCode=FE619, only 19 are listed:

youtube-dl --list-subs https://www.vlive.tv/video/3697?channelCode=FE619
[vlive] 3697: Downloading webpage
[vlive] 3697: Downloading JSON metadata
Available subtitles for 3697:
Language formats
ro_RO    vtt
en_US    vtt
in_ID    vtt
zh_CN    vtt
el_GR    vtt
ko_KR    vtt
pt_BR    vtt
ru_RU    vtt
hu_HU    vtt
de_DE    vtt
ar_AE    vtt
th_TH    vtt
vi_VN    vtt
fr_FR    vtt
pl_PL    vtt
it_IT    vtt
es_ES    vtt
zh_TW    vtt
tr_TR    vtt

vlive_subtitles_3697

Help with this issue would be appreciated as I'm trying to download several hundred videos and would like the option of which en_US subtitle track to download (or perhaps the ability to include both.)

wiznaibus commented 5 years ago

Did some poking around (I'm not familiar with Python so please excuse any mistakes) and I discovered that multiple subtitle tracks for the same language on a V LIVE video are being overwritten in the subtitles variable since the key is simply [lang]:

In vlive.py:

subtitles = {}
for caption in playinfo.get('captions', {}).get('list', []):
    lang = dict_get(caption, ('locale', 'language', 'country', 'label'))
    if lang and caption.get('source'):
        subtitles[lang] = [{
            'ext': 'vtt',
            'url': caption['source']}]

The only difference in these two subtitle tracks is in the source URL where one is labeled en_US and the other en_US_fan:

Using print(caption):

{'language': 'en', 'country': 'US', 'locale': 'en_US', 'label': 'English', 'source': 'http://caption.rmcnmv.naver.net/globalv/global_meta/read/global_v_2015_11_28_5/531C05F31314F3CC8C33FE1493383533C04_muploader_b_360P_640_1228_128-1448745362393_en_US.vtt?__gda__=[private]'}
{'language': 'en', 'country': 'US', 'locale': 'en_US', 'label': 'English', 'source': 'http://caption.rmcnmv.naver.net/globalv/global_meta/read/global_v_2018_01_14_5/1962968b-f8c0-11e7-8d4e-3ca82a22c1e9-1515888960141_en_US_fan.vtt?__gda__=[private]'}

Maybe the source URL could be parsed to determine if a track is an official subtitle or fansub and that distinction could be made part of the key?

I'm also going to try and determine whether V LIVE allows multiple fan-submitted subtitle tracks for a language, or if each language is limited to one official subtitle track and one fan-submitted track.

Please let me know if I should submit a new issue for this as it involves V LIVE specifically and not Crunchyroll, even though it is related to the same core issue of handling multiple subtitle files for the same language.

Thank you!

--

Edit: Found a quick and dirty solution to my V LIVE issue! It's by no means complete (especially since it's hardcoded for English), but maybe it can help inspire a future bug-fix.

In vlive.py:

subtitles = {}
for caption in playinfo.get('captions', {}).get('list', []):
    lang = dict_get(caption, ('locale', 'language', 'country', 'label'))
    if 'fan.vtt' in caption['source']:
        lang += '_fan'
    if lang and caption.get('source'):
        subtitles[lang] = [{
            'ext': 'vtt',
            'url': caption['source']}]

In ffmpeg.py:

for (i, lang) in enumerate(sub_langs):
    opts.extend(['-map', '%d:0' % (i + 1)])
    if (lang == 'en_US_fan'):
        lang_code = 'English (Fan)'
    elif (lang == 'en_US'):
        lang_code = 'English'
    else:
        lang_code = ISO639Utils.short2long(lang) or lang
    opts.extend(['-metadata:s:s:%d' % i, 'language=%s' % lang, '-metadata:s:s:%d' % i, 'handler=%s' % lang_code])

(Adding the handler option to the ffmpeg command is necessary for it to label the two tracks correctly, otherwise they'll both be labeled "English".)

Then, the subtitles can be included using --sub-lang en_US,en_US_fan or you can pick one or the other.

darkhelmet2016 commented 4 years ago

Think the problem with the alternate English sub for crunchy is that it might be overwriting the sub and maybe adding a line that it a sub exist when downloading it needs a line in the code where it download the second English sub and annotate it as say English (US) alt.ass file.

Something like. Writing video subtitles to: Ace Attorney Season 2 Episode 16 – Turnabout Beginnings — Last Trial-782001.enUS.ass Writing video subtitles to: Ace Attorney Season 2 Episode 16 – Turnabout Beginnings — Last Trial-782001.enUS (alt).ass

asc6 commented 4 years ago

I know this is an old issue, but it appears that this issue still exists on the latest version 2020.03.08. I'm having the same issue on a different Ace Attorney episode from crunchyroll. --list-subs shows two English versions, but only one ever gets downloaded.

Downloads$ youtube-dl --list-subs --skip-download https://www.crunchyroll.com/ace-attorney/episode-10-northward-turnabout-express-1st-trial-777744                         
[crunchyroll] 777744: Downloading webpage
[crunchyroll] 777744: Downloading adaptive_hls-audio-jaJP-hardsub-esES information
[crunchyroll] 777744: Downloading adaptive_hls-audio-jaJP-hardsub-deDE information
[crunchyroll] 777744: Downloading adaptive_hls-audio-jaJP-hardsub-frFR information
[crunchyroll] 777744: Downloading adaptive_hls-audio-jaJP-hardsub-esLA information
[crunchyroll] 777744: Downloading adaptive_hls-audio-jaJP information
[crunchyroll] 777744: Downloading adaptive_hls-audio-jaJP-hardsub-ruRU information
[crunchyroll] 777744: Downloading adaptive_hls-audio-jaJP-hardsub-arME information
[crunchyroll] 777744: Downloading adaptive_hls-audio-jaJP-hardsub-ptBR information
[crunchyroll] 777744: Downloading adaptive_hls-audio-jaJP-hardsub-enUS information
[crunchyroll] 777744: Downloading adaptive_hls-audio-jaJP-hardsub-itIT information
[crunchyroll] 777744: Downloading media info
Available subtitles for 777744:
Language formats
ruRU     ass
esES     ass
itIT     ass
deDE     ass
esLA     ass
enUS     ass, ass
frFR     ass
arME     ass
ptBR     ass

Downloads$ youtube-dl -v --write-sub  --sub-lang enUS --skip-download https://www.crunchyroll.com/ace-attorney/episode-10-northward-turnabout-express-1st-trial-777744 
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'--write-sub', u'--sub-lang', u'enUS', u'--skip-download', u'https://www.crunchyroll.com/ace-attorney/episode-10-northward-turnabout-express-1st-trial-777744']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2020.03.08
[debug] Python version 2.7.16 (CPython) - Darwin-19.3.0-x86_64-i386-64bit
[debug] exe versions: none
[debug] Proxy map: {}
[crunchyroll] 777744: Downloading webpage
[crunchyroll] 777744: Downloading adaptive_hls-audio-jaJP-hardsub-esES information
[crunchyroll] 777744: Downloading adaptive_hls-audio-jaJP-hardsub-deDE information
[crunchyroll] 777744: Downloading adaptive_hls-audio-jaJP-hardsub-frFR information
[crunchyroll] 777744: Downloading adaptive_hls-audio-jaJP-hardsub-esLA information
[crunchyroll] 777744: Downloading adaptive_hls-audio-jaJP information
[crunchyroll] 777744: Downloading adaptive_hls-audio-jaJP-hardsub-ruRU information
[crunchyroll] 777744: Downloading adaptive_hls-audio-jaJP-hardsub-arME information
[crunchyroll] 777744: Downloading adaptive_hls-audio-jaJP-hardsub-ptBR information
[crunchyroll] 777744: Downloading adaptive_hls-audio-jaJP-hardsub-enUS information
[crunchyroll] 777744: Downloading adaptive_hls-audio-jaJP-hardsub-itIT information
[crunchyroll] 777744: Downloading media info
[debug] Default format spec: best/bestvideo+bestaudio
[info] Writing video subtitles to: Ace Attorney Season 2 Episode 10 – Northward, Turnabout Express — 1st Trial-777744.enUS.ass

I agree with darkhelmet2016's point, that having some sort of "(alt)" text would be great for multiple versions.