search

ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
131.91k stars 10k forks source link

Unable to download VOD from twitcasting.tv #29248

Open pyntox opened 3 years ago

pyntox commented 3 years ago

Checklist

Verbose log

> .\youtube-dl.exe --verbose --add-header "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36" --add-header "Origin: https://twitcasting.tv" https://dl01.twitcasting.tv/tc.vod/v/686822522.0.2-1623256184-1623284984-8c7d244a-a574077b2e4b8308/fmp4/index.m3u8
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--verbose', '--add-header', 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36', '--add-header', 'Origin: https://twitcasting.tv', 'https://dl01.twitcasting.tv/tc.vod/v/686822522.0.2-1623256184-1623284984-8c7d244a-a574077b2e4b8308/fmp4/index.m3u8']
[debug] Adding header from command line option User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36
[debug] Adding header from command line option Origin: https://twitcasting.tv
[debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252
[debug] youtube-dl version 2021.06.06
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.19041
[debug] exe versions: ffmpeg 4.4-full_build-www.gyan.dev, ffprobe 4.4-full_build-www.gyan.dev
[debug] Proxy map: {}
[generic] index: Requesting header
WARNING: Could not send HEAD request to https://dl01.twitcasting.tv/tc.vod/v/686822522.0.2-1623256184-1623284984-8c7d244a-a574077b2e4b8308/fmp4/index.m3u8: HTTP Error 502: Bad Gateway
[generic] index: Downloading webpage
ERROR: Unable to download webpage: HTTP Error 502: Bad Gateway (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpkqxnwl31\build\youtube_dl\extractor\common.py", line 634, in _request_webpage
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpkqxnwl31\build\youtube_dl\YoutubeDL.py", line 2288, in urlopen
  File "C:\Python\Python34\lib\urllib\request.py", line 470, in open
  File "C:\Python\Python34\lib\urllib\request.py", line 580, in http_response
  File "C:\Python\Python34\lib\urllib\request.py", line 508, in error
  File "C:\Python\Python34\lib\urllib\request.py", line 442, in _call_chain
  File "C:\Python\Python34\lib\urllib\request.py", line 588, in http_error_default

Description

I checked other similar issues (https://github.com/yt-dlp/yt-dlp/issues/220 in particular) and tried these two commands (also with double quotes for the .m3u8 link):

> .\youtube-dl.exe --verbose --user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36" --add-header "Origin: https://twitcasting.tv" https://dl01.twitcasting.tv/tc.vod/v/686822522.0.2-1623256184-1623284984-8c7d244a-a574077b2e4b8308/fmp4/index.m3u8
> .\youtube-dl.exe --verbose --add-header "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36" --add-header "Origin: https://twitcasting.tv" -i https://dl01.twitcasting.tv/tc.vod/v/686822522.0.2-1623256184-1623284984-8c7d244a-a574077b2e4b8308/fmp4/index.m3u8

Both of them led to "Error 502: Bad Gateway"

I also used this:

> .\ffmpeg -re -user_agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36" -headers "Origin: https://twitcasting.tv" -n -i "https://dl01.twitcasting.tv/tc.vod/v/686822522.2.2-1623256184-1623284984-8c7d244a-6736bdbb96e2e570/fmp4/index.m3u8" -c copy -bsf:a aac_adtstoasc -f mp4 test_nanoch_1.mp4

The video I'm trying to download is this one: https://twitcasting.tv/angelnano1004/movie/686822522

ghost commented 3 years ago

If you've extracted m3u8 url from the browser, try adding --referer https://twitcasting.tv/

And I did another way.

1. Open the page and save cookie file.

curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36" https://twitcasting.tv/angelnano1004/movie/686822522 -c cookie.txt >/dev/null

2. Open the page again using the cookie file and extract m3u8 url.

curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36" https://twitcasting.tv/angelnano1004/movie/686822522 -b cookie.txt | gsed -n "/.*data-movie-playlist='/{s///;s/'$//;p;}" | jq -r ".\"2\"[0].source.url"

output: (changes every time)

https://dl01.twitcasting.tv/tc.vod/v/686822522.0.2-1624474988-1624503788-8e4e3a28-81eb8c2c030a7352/fmp4/index.m3u8

3. Open m3u8 url with the same user-agent and give referer. (ffmpeg will be used for download anyways.)

youtube-dl --user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36" --referer https://twitcasting.tv/ -o "686822522.%(ext)s" https://dl01.twitcasting.tv/tc.vod/v/686822522.0.2-1624474988-1624503788-8e4e3a28-81eb8c2c030a7352/fmp4/index.m3u8

It was also possible to use fake cookie.

1'. Make fake cookie.

python -c "import uuid; print(uuid.uuid4().hex)"

output: (changes every time)

fda26e4110f54f0abf77f92761f2e49d

Use this value in the above #2. (as -b did=fda26e4110f54f0abf77f92761f2e49d instead of -b cookie.txt)