tver.jp no longer works after website redesign

sabuaka18 commented 2 years ago

Checklist

[ x] I'm reporting a broken site support
[ x] I've verified that I'm running youtube-dl version 2021.12.17
[ x] I've checked that all provided URLs are alive and playable in a browser
[ x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[ x] I've searched the bugtracker for similar issues including closed ones

Verbose log

youtube-dl https://tver.jp/corner/f0103888-v --ignore-config
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'https://tver.jp/corner/f0103888', u'-v', u'--ignore-config']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 2.7.16 (CPython) - Linux-4.19.0-20-cloud-amd64-x86_64-with-debian-10.12
[debug] exe versions: ffmpeg 4.1.8-0, ffprobe 4.1.8-0, rtmpdump 2.4
[debug] Proxy map: {}
[TVer] Downloading JSON metadata
ERROR: Unable to download JSON metadata: HTTP Error 404: Not Found (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 634, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2288, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 473, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

Description

Tver.jp has had a major overhaul of the website the old https://tver.jp/api/access_token.php no longer works. It uses somthing like https://platform-api.tver.jp/v2/api/platform_users/info?platform_uid=xxxxxxxxxxxxxxxx&platform_token=xxxxxxxxxxxxxxxx now the old https://tver.jp/corner/xxxxx or https://tver.jp/feature/xxxxxx now redirects to https://tver.jp/episodes/xxxxxxx for example: https://tver.jp/corner/f0103888 now redirects to https://tver.jp/episodes/ep6f16g26p this seems like to be the example of m3u8 url for the url above https://manifest.prod.boltdns.net/manifest/v1/hls/v4/aes128/3971130137001/xxxxx-xxxx-xxxx-xxxx-xxxxxxx/10s/master.m3u8?behavior_id=xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxx&fastly_token=xxxxxxxxxxxxxxxxxxxxxxxx

foomin10 commented 2 years ago

not [video/videoID] but [video/videoRefID]

suzukitakeomi commented 2 years ago

TVer(20220401ver) URL convert sttep.

Step1 - Get TVerID https://tver.jp/episodes/(tverid)

Step2 - Get accountID and videoRefID https://statics.tver.jp/content/episode/(tverid).json?v=5 or https://statics.tver.jp/content/episode/(tverid).json?v=8 get[video/accountID] get[video/videoID] -> goto Step 4 get[video/videoRefID] -> goto Step 3

Step3 - Get videoId(Step2: [video/videoRefID] only) https://edge.api.brightcove.com/playback/v1/accounts/[video/accountID]/videos/ref%3A[video/videoRefID] get[video/videoID]

Step4 - Get Video URL http://players.brightcove.net/[video/accountID]/default_default/index.html?videoId=[video/videoID]

dirkf commented 2 years ago

Brightcove is returning 404 from the UK with the videoID for https://tver.jp/episodes/ep6f16g26p where there's no videoRefID in the video object of the JSON.

suzukitakeomi commented 2 years ago

Brightcove is returning 404 from the UK with the videoID for https://tver.jp/episodes/ep6f16g26p where there's no videoRefID in the video object of the JSON.

ex) https://tver.jp/episodes/ep6f16g26p Step1 get[ https://statics.tver.jp/content/episode/ep6f16g26p.json?v=7 ] Step2 json check [video/accountid] => 3971130137001 [video/videoid] => 6302378806001 [video/videorefid] => (undefined) Step3 Skip.(videoid ok) Step4 get[ http://players.brightcove.net/3971130137001/default_default/index.html?videoId=6302378806001 ]

Connection from Japan

suzukitakeomi commented 2 years ago

(tverid) -> Video URL Converter.

PHP Code <?php $response = ''; $r = 0; $conn = curl_init(); curl_setopt($conn, CURLOPT_CONNECTTIMEOUT, 2); curl_setopt($conn, CURLOPT_RETURNTRANSFER, TRUE); curl_setopt($conn, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Safari/537.36'); curl_setopt($conn, CURLOPT_URL, 'https://statics.tver.jp/content/episode/'.$episodes.'.json?v=7'); curl_setopt($conn, CURLOPT_HTTPHEADER, [ 'origin: https://tver.jp', 'referer: https://tver.jp/', 'sec-fetch-mode: cors' ]); $response = curl_exec($conn); $info = curl_getinfo($conn); if(isset($info['http_code']) && $info['http_code'] == 200){ $r = 1; } curl_close($conn); if($r){ $tver['accountid'] = ''; $tver['videoid'] = ''; $tver['videorefid'] = ''; $json = json_decode($response, 1); if(isset($json['video']['accountID']) && $json['video']['accountID']){ $tver['accountid'] = $json['video']['accountID']; } if(isset($json['video']['videoID']) && $json['video']['videoID']){ $tver['videoid'] = $json['video']['videoID']; } if(isset($json['video']['videoRefID']) && $json['video']['videoRefID']){ $tver['videorefid'] = $json['video']['videoRefID']; } if($tver['videoid'] == '' && $tver['videorefid'] != ''){ $tver['videoid'] = videorefid2videoid($tver['accountid'], $tver['videorefid']); } if($tver['videoid'] == ''){ /*Error*/ }else{ printf('http://players.brightcove.net/%s/default_default/index.html?videoId=%s', $tver['accountid'], $tver['videoid']); } }else{ /*Error*/ } function videorefid2videoid($accountid, $videorefid){ $videoid = ''; $response = ''; $r = 0; $conn = curl_init(); curl_setopt($conn, CURLOPT_CONNECTTIMEOUT, 2); curl_setopt($conn, CURLOPT_RETURNTRANSFER, TRUE); curl_setopt($conn, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Safari/537.36'); curl_setopt($conn, CURLOPT_URL, 'https://edge.api.brightcove.com/playback/v1/accounts/'.$accountid.'/videos/ref%3A'.$videorefid); curl_setopt($conn, CURLOPT_HTTPHEADER, [ 'accept: application/json;pk=BCpkA.................................' ]); $response = curl_exec($conn); $info = curl_getinfo($conn); if(isset($info['http_code']) && $info['http_code'] == 200){ $r = 1; } curl_close($conn); if($r){ $json = json_decode($response, 1); if(isset($json['id']) && $json['id'] != ''){ $videoid = $json['id']; } } return($videoid); } ?>

Lesmiscore commented 2 years ago

As I submitted at https://github.com/yt-dlp/yt-dlp/pull/3268 , BrightcoveNewIE can handle both videoID and videoRefID, when parameters are appropriately passed.

qbabe commented 2 years ago

TVer(20220401ver) URL convert sttep.

Step1 - Get TVerID https://tver.jp/episodes/(tverid)

Step2 - Get accountID and videoRefID https://statics.tver.jp/content/episode/(tverid).json?v=5 or https://statics.tver.jp/content/episode/(tverid).json?v=8 get[video/accountID] get[video/videoID] -> goto Step 4 get[video/videoRefID] -> goto Step 3

Step3 - Get videoId(Step2: [video/videoRefID] only) https://edge.api.brightcove.com/playback/v1/accounts/[video/accountID]/videos/ref%3A[video/videoRefID] get[video/videoID]

Step4 - Get Video URL http://players.brightcove.net/[video/accountID]/default_default/index.html?videoId=[video/videoID]

Brightcove is returning 404 for https://tver.jp/episodes/ep5y3111ol where there's only videoRefID in the video object of the JSON. ex) https://tver.jp/episodes/ep5y3111ol Step1 get[ https://statics.tver.jp/content/episode/ep5y3111ol.json?v=7 ] Step2 json check [video/accountid] => 4031511847001 [video/videoid] => (undefined) [video/videorefid] => 20144_2000609_1000017963 Step3 - Get videoId(Step2: [video/videoRefID] only) https://edge.api.brightcove.com/playback/v1/accounts/4031511847001/videos/ref%3A20144_2000609_1000017963

returning "error_code": "INVALID_POLICY_KEY", "message": "Request policy key is missing or invalid"

gsool9988 commented 2 years ago

It seems VideoID is provided for a few contents - news, sports, etc. Then, Step4 works well. But in the case of VideoRefID , Step3 is not working and occurs the error.

error_code "INVALID_POLICY_KEY" message "Request policy key is missing or invalid"

dirkf commented 2 years ago

A PR based on yt-dlp #3268 will appear shortly.

gsool9988 commented 2 years ago

The solution you sugested works well now. Thanx for your information.

qbabe commented 2 years ago

The solution you sugested works well now. Thanx for your information.

Did you find the solution of "videoRefID" problem??

gsool9988 commented 2 years ago

dirkf uploaded fixed extractor for Tver. https://github.com/dirkf/youtube-dl/blob/7c254e260f348fe236c54580bf416a40f8fe0063/youtube_dl/extractor/tver.py

To put in a nut shell, in the case of VideorefID provided, then VideoID = 'ref:'+(VideorefID) For your example, [video/videorefid] => 20144_2000609_1000017963 then, VideoID = ref:20144_2000609_1000017963 and apply the VideoID to Step4,

dirkf commented 2 years ago

Please load and test this PR if you can, and report in the PR thread.

ngocbae98 commented 2 years ago

guys, it's can update for the windows version?

ytdl-org / youtube-dl