search

blackjack4494 / yt-dlc

media downloader and library for various sites.
The Unlicense
2.9k stars 365 forks source link

[Broken] Twitch.tv via yt-dlc library calls / Tubeup #76

Closed vxbinaca closed 3 years ago

vxbinaca commented 3 years ago

Checklist

Verbose log

tubeup --debug --use-download-archive --metadata=collection:twitchstreams "https://www.twitch.tv/videos/787134928"

[DEBUG] 2020-10-31 05:08:58,227 - tubeup.TubeUp - DEBUG - [debug] Loading archive file '/home/vxbinaca/.tubeup/.ytdlarchive'
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dlc version 2020.10.31
[debug] Python version 3.7.5 (CPython) - Linux-4.15.0-122-generic-x86_64-with-Ubuntu-18.04-bionic
[debug] exe versions: ffmpeg 3.4.8, ffprobe 3.4.8, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {}
[DEBUG] 2020-10-31 05:09:00,598 - tubeup.TubeUp - DEBUG - [twitch:vod] 787134928: Downloading video info JSON
[DEBUG] 2020-10-31 05:09:00,922 - tubeup.TubeUp - DEBUG - [twitch:vod] 787134928: Downloading vod access token
[DEBUG] 2020-10-31 05:09:01,295 - tubeup.TubeUp - DEBUG - [twitch:vod] 787134928: Downloading m3u8 information
[DEBUG] 2020-10-31 05:09:02,091 - tubeup.TubeUp - DEBUG - [debug] Default format spec: bestvideo+bestaudio/best
[DEBUG] 2020-10-31 05:09:02,166 - tubeup.TubeUp - DEBUG - 🎃 !HALLOWEEK 🎃 Until Dawn - FIRST TIME 🖤 !goal
[DEBUG] 2020-10-31 05:09:02,167 - tubeup.TubeUp - DEBUG - {"id": "v787134928", "title": "\ud83c\udf83 !HALLOWEEK \ud83c\udf83 Until Dawn - FIRST TIME \ud83d\udda4 !goal", "description": null, "duration": 23589, "thumbnails": [{"url": "https://static-cdn.jtvnw.net/cf_vods/d2nvs31859zcd8/688bf5d61347ad152d24_spoonee_40286011342_1604106121//thumb/thumb0-80x45.jpg", "preference": 0, "id": "0"}, {"url": "https://static-cdn.jtvnw.net/cf_vods/d2nvs31859zcd8/688bf5d61347ad152d24_spoonee_40286011342_1604106121//thumb/thumb0-320x180.jpg", "preference": 1, "id": "1"}, {"url": "https://static-cdn.jtvnw.net/cf_vods/d2nvs31859zcd8/688bf5d61347ad152d24_spoonee_40286011342_1604106121//thumb/thumb0-640x360.jpg", "preference": 2, "id": "2"}], "uploader": "spoonee", "uploader_id": "spoonee", "timestamp": 1604106130, "view_count": 2054, "is_live": false, "formats": [{"format_id": "Audio_Only", "url": "https://dqrpb9wgowsf5.cloudfront.net/688bf5d61347ad152d24_spoonee_40286011342_1604106121/audio_only/index-dvr.m3u8", "manifest_url": "https://usher.ttvnw.net/vod/787134928.m3u8?allow_source=true&allow_audio_only=true&allow_spectre=true&player=twitchweb&playlist_include_framerate=true&nauth=%7B%22authorization%22%3A%7B%22forbidden%22%3Afalse%2C%22reason%22%3A%22%22%7D%2C%22chansub%22%3A%7B%22restricted_bitrates%22%3A%5B%5D%7D%2C%22device_id%22%3Anull%2C%22expires%22%3A1604207341%2C%22https_required%22%3Afalse%2C%22privileged%22%3Afalse%2C%22user_id%22%3Anull%2C%22version%22%3A2%2C%22vod_id%22%3A787134928%7D&nauthsig=3b045fbeab0c020cd6d39b06258f1dc3d9b252db", "tbr": 214.766, "ext": "mp4", "fps": null, "protocol": "m3u8_native", "preference": null, "vcodec": "none", "acodec": "mp4a.40.2", "format": "Audio_Only - audio only", "http_headers": {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.57 Safari/537.36", "Accept-Charset": "ISO-8859-1,utf-8;q=0.7,*;q=0.7", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Encoding": "gzip, deflate", "Accept-Language": "en-us,en;q=0.5"}}, {"format_id": "160p", "url": "https://dqrpb9wgowsf5.cloudfront.net/688bf5d61347ad152d24_spoonee_40286011342_1604106121/160p30/index-dvr.m3u8", "manifest_url": "https://usher.ttvnw.net/vod/787134928.m3u8?allow_source=true&allow_audio_only=true&allow_spectre=true&player=twitchweb&playlist_include_framerate=true&nauth=%7B%22authorization%22%3A%7B%22forbidden%22%3Afalse%2C%22reason%22%3A%22%22%7D%2C%22chansub%22%3A%7B%22restricted_bitrates%22%3A%5B%5D%7D%2C%22device_id%22%3Anull%2C%22expires%22%3A1604207341%2C%22https_required%22%3Afalse%2C%22privileged%22%3Afalse%2C%22user_id%22%3Anull%2C%22version%22%3A2%2C%22vod_id%22%3A787134928%7D&nauthsig=3b045fbeab0c020cd6d39b06258f1dc3d9b252db", "tbr": 286.653, "ext": "mp4", "fps": 29.999, "protocol": "m3u8_native", "preference": null, "width": 284, "height": 160, "vcodec": "avc1.4D400C", "acodec": "mp4a.40.2", "format": "160p - 284x160", "http_headers": {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.57 Safari/537.36", "Accept-Charset": "ISO-8859-1,utf-8;q=0.7,*;q=0.7", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Encoding": "gzip, deflate", "Accept-Language": "en-us,en;q=0.5"}}, {"format_id": "360p", "url": "https://dqrpb9wgowsf5.cloudfront.net/688bf5d61347ad152d24_spoonee_40286011342_1604106121/360p30/index-dvr.m3u8", "manifest_url": "https://usher.ttvnw.net/vod/787134928.m3u8?allow_source=true&allow_audio_only=true&allow_spectre=true&player=twitchweb&playlist_include_framerate=true&nauth=%7B%22authorization%22%3A%7B%22forbidden%22%3Afalse%2C%22reason%22%3A%22%22%7D%2C%22chansub%22%3A%7B%22restricted_bitrates%22%3A%5B%5D%7D%2C%22device_id%22%3Anull%2C%22expires%22%3A1604207341%2C%22https_required%22%3Afalse%2C%22privileged%22%3Afalse%2C%22user_id%22%3Anull%2C%22version%22%3A2%2C%22vod_id%22%3A787134928%7D&nauthsig=3b045fbeab0c020cd6d39b06258f1dc3d9b252db", "tbr": 695.622, "ext": "mp4", "fps": 29.999, "protocol": "m3u8_native", "preference": null, "width": 640, "height": 360, "vcodec": "avc1.4D401E", "acodec": "mp4a.40.2", "format": "360p - 640x360", "http_headers": {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.57 Safari/537.36", "Accept-Charset": "ISO-8859-1,utf-8;q=0.7,*;q=0.7", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Encoding": "gzip, deflate", "Accept-Language": "en-us,en;q=0.5"}}, {"format_id": "480p", "url": "https://dqrpb9wgowsf5.cloudfront.net/688bf5d61347ad152d24_spoonee_40286011342_1604106121/480p30/index-dvr.m3u8", "manifest_url": "https://usher.ttvnw.net/vod/787134928.m3u8?allow_source=true&allow_audio_only=true&allow_spectre=true&player=twitchweb&playlist_include_framerate=true&nauth=%7B%22authorization%22%3A%7B%22forbidden%22%3Afalse%2C%22reason%22%3A%22%22%7D%2C%22chansub%22%3A%7B%22restricted_bitrates%22%3A%5B%5D%7D%2C%22device_id%22%3Anull%2C%22expires%22%3A1604207341%2C%22https_required%22%3Afalse%2C%22privileged%22%3Afalse%2C%22user_id%22%3Anull%2C%22version%22%3A2%2C%22vod_id%22%3A787134928%7D&nauthsig=3b045fbeab0c020cd6d39b06258f1dc3d9b252db", "tbr": 1413.485, "ext": "mp4", "fps": 29.999, "protocol": "m3u8_native", "preference": null, "width": 852, "height": 480, "vcodec": "avc1.4D401E", "acodec": "mp4a.40.2", "format": "480p - 852x480", "http_headers": {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.57 Safari/537.36", "Accept-Charset": "ISO-8859-1,utf-8;q=0.7,*;q=0.7", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Encoding": "gzip, deflate", "Accept-Language": "en-us,en;q=0.5"}}, {"format_id": "720p-0", "url": "https://dqrpb9wgowsf5.cloudfront.net/688bf5d61347ad152d24_spoonee_40286011342_1604106121/720p30/index-dvr.m3u8", "manifest_url": "https://usher.ttvnw.net/vod/787134928.m3u8?allow_source=true&allow_audio_only=true&allow_spectre=true&player=twitchweb&playlist_include_framerate=true&nauth=%7B%22authorization%22%3A%7B%22forbidden%22%3Afalse%2C%22reason%22%3A%22%22%7D%2C%22chansub%22%3A%7B%22restricted_bitrates%22%3A%5B%5D%7D%2C%22device_id%22%3Anull%2C%22expires%22%3A1604207341%2C%22https_required%22%3Afalse%2C%22privileged%22%3Afalse%2C%22user_id%22%3Anull%2C%22version%22%3A2%2C%22vod_id%22%3A787134928%7D&nauthsig=3b045fbeab0c020cd6d39b06258f1dc3d9b252db", "tbr": 2263.174, "ext": "mp4", "fps": 29.999, "protocol": "m3u8_native", "preference": null, "width": 1280, "height": 720, "vcodec": "avc1.4D401F", "acodec": "mp4a.40.2", "format": "720p-0 - 1280x720", "http_headers": {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.57 Safari/537.36", "Accept-Charset": "ISO-8859-1,utf-8;q=0.7,*;q=0.7", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Encoding": "gzip, deflate", "Accept-Language": "en-us,en;q=0.5"}}, {"format_id": "720p-1", "url": "https://dqrpb9wgowsf5.cloudfront.net/688bf5d61347ad152d24_spoonee_40286011342_1604106121/chunked/index-dvr.m3u8", "manifest_url": "https://usher.ttvnw.net/vod/787134928.m3u8?allow_source=true&allow_audio_only=true&allow_spectre=true&player=twitchweb&playlist_include_framerate=true&nauth=%7B%22authorization%22%3A%7B%22forbidden%22%3Afalse%2C%22reason%22%3A%22%22%7D%2C%22chansub%22%3A%7B%22restricted_bitrates%22%3A%5B%5D%7D%2C%22device_id%22%3Anull%2C%22expires%22%3A1604207341%2C%22https_required%22%3Afalse%2C%22privileged%22%3Afalse%2C%22user_id%22%3Anull%2C%22version%22%3A2%2C%22vod_id%22%3A787134928%7D&nauthsig=3b045fbeab0c020cd6d39b06258f1dc3d9b252db", "tbr": 3291.825, "ext": "mp4", "fps": 29.999, "protocol": "m3u8_native", "preference": null, "width": 1280, "height": 720, "vcodec": "avc1.64001F", "acodec": "mp4a.40.2", "quality": 10, "format_note": "Source", "format": "720p-1 - 1280x720 (Source)", "http_headers": {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.57 Safari/537.36", "Accept-Charset": "ISO-8859-1,utf-8;q=0.7,*;q=0.7", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Encoding": "gzip, deflate", "Accept-Language": "en-us,en;q=0.5"}}], "subtitles": {"rechat": [{"url": "https://api.twitch.tv/v5/videos/787134928/comments?client_id=kimne78kx3ncx6brgo4mv6wki5h1ko", "ext": "json"}]}, "extractor": "twitch:vod", "webpage_url": "https://www.twitch.tv/videos/787134928", "webpage_url_basename": "787134928", "extractor_key": "TwitchVod", "playlist": null, "playlist_index": null, "thumbnail": "https://static-cdn.jtvnw.net/cf_vods/d2nvs31859zcd8/688bf5d61347ad152d24_spoonee_40286011342_1604106121//thumb/thumb0-640x360.jpg", "display_id": "v787134928", "upload_date": "20201031", "requested_subtitles": {"rechat": {"url": "https://api.twitch.tv/v5/videos/787134928/comments?client_id=kimne78kx3ncx6brgo4mv6wki5h1ko", "ext": "json"}}, "format_id": "720p-1", "url": "https://dqrpb9wgowsf5.cloudfront.net/688bf5d61347ad152d24_spoonee_40286011342_1604106121/chunked/index-dvr.m3u8", "manifest_url": "https://usher.ttvnw.net/vod/787134928.m3u8?allow_source=true&allow_audio_only=true&allow_spectre=true&player=twitchweb&playlist_include_framerate=true&nauth=%7B%22authorization%22%3A%7B%22forbidden%22%3Afalse%2C%22reason%22%3A%22%22%7D%2C%22chansub%22%3A%7B%22restricted_bitrates%22%3A%5B%5D%7D%2C%22device_id%22%3Anull%2C%22expires%22%3A1604207341%2C%22https_required%22%3Afalse%2C%22privileged%22%3Afalse%2C%22user_id%22%3Anull%2C%22version%22%3A2%2C%22vod_id%22%3A787134928%7D&nauthsig=3b045fbeab0c020cd6d39b06258f1dc3d9b252db", "tbr": 3291.825, "ext": "mp4", "fps": 29.999, "protocol": "m3u8_native", "preference": null, "width": 1280, "height": 720, "vcodec": "avc1.64001F", "acodec": "mp4a.40.2", "quality": 10, "format_note": "Source", "format": "720p-1 - 1280x720 (Source)", "http_headers": {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.57 Safari/537.36", "Accept-Charset": "ISO-8859-1,utf-8;q=0.7,*;q=0.7", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Encoding": "gzip, deflate", "Accept-Language": "en-us,en;q=0.5"}, "fulltitle": "\ud83c\udf83 !HALLOWEEK \ud83c\udf83 Until Dawn - FIRST TIME \ud83d\udda4 !goal", "_filename": "/home/vxbinaca/.tubeup/downloads/v787134928.mp4"}
[DEBUG] 2020-10-31 05:09:02,168 - tubeup.TubeUp - WARNING - There's no description to write.
[DEBUG] 2020-10-31 05:09:02,169 - tubeup.TubeUp - WARNING - There are no annotations to write.
[DEBUG] 2020-10-31 05:09:02,169 - tubeup.TubeUp - DEBUG - [info] Writing video subtitles to: /home/vxbinaca/.tubeup/downloads/v787134928.rechat.json
[DEBUG] 2020-10-31 05:09:02,170 - tubeup.TubeUp - DEBUG - [debug] Invoking downloader on 'https://api.twitch.tv/v5/videos/787134928/comments?client_id=kimne78kx3ncx6brgo4mv6wki5h1ko'
[DEBUG] 2020-10-31 05:09:02,172 - tubeup.TubeUp - ERROR - ERROR: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
[DEBUG] 2020-10-31 05:09:02,173 - tubeup.TubeUp - ERROR - Traceback (most recent call last):
  File "/home/vxbinaca/.local/lib/python3.7/site-packages/youtube_dlc/YoutubeDL.py", line 846, in extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "/home/vxbinaca/.local/lib/python3.7/site-packages/youtube_dlc/YoutubeDL.py", line 901, in process_ie_result
    return self.process_video_result(ie_result, download=download)
  File "/home/vxbinaca/.local/lib/python3.7/site-packages/youtube_dlc/YoutubeDL.py", line 1696, in process_video_result
    self.process_info(new_info)
  File "/home/vxbinaca/.local/lib/python3.7/site-packages/youtube_dlc/YoutubeDL.py", line 1894, in process_info
    dl(sub_filename, sub_info, subtitle=True)
  File "/home/vxbinaca/.local/lib/python3.7/site-packages/youtube_dlc/YoutubeDL.py", line 1866, in dl
    return fd.download(name, info, subtitle)
  File "/home/vxbinaca/.local/lib/python3.7/site-packages/youtube_dlc/downloader/common.py", line 370, in download
    int(sleep_interval_sub)))
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'

Description

Something in 2020.10.31 broke extraction of Twitch channels/videos, my software depends on yours now, and switching back to 2020.10.26 resolved the issue for me.

Using your software alone works, but something with the latest version and my software is breaking extraction of Twitch video and video IDs from channels.

We call use these library calls:

            'outtmpl': os.path.join(self.dir_path['downloads'],
                                    self.output_template),
            'restrictfilenames': True,
            'quiet': not self.verbose,
            'verbose': self.verbose,
            'progress_with_newline': True,
            'forcetitle': True,
            'continuedl': True,
            'retries': 9001,
            'fragment_retries': 9001,
            'forcejson': True,
            'writeinfojson': True,
            'writedescription': True,
            'writethumbnail': True,                                                                                                                   
            'writeannotations': True,
            'writesubtitles': True,
            'allsubtitles': True,
            'ignoreerrors': True,  # Geo-blocked,
                                   # copyrighted/private/deleted
                                   # will be printed to STDOUT and channel
                                   # ripping will  continue uninterupted,
                                   # use with verbose off
            'fixup': 'warn',  # Slightly more verbosity for debugging
                              # problems
            'nooverwrites': True,  # Don't touch what's already been
                                   # downloaded speeds things
            'consoletitle': True,   # Download percentage in console title
            'prefer_ffmpeg': True,  # `ffmpeg` is better than `avconv`,
                                    # let's prefer it's use
            # Warns on out of date youtube-dl script, helps debugging for
            # youtube-dl devs
            'call_home': False,
            'logger': self.logger,
            'progress_hooks': [ydl_progress_hook]
blackjack4494 commented 3 years ago

@vxbinaca are you still experiencing problems?

vxbinaca commented 3 years ago

Nope, apologies about forgetting to close this.