Viu OTT extractor error (KeyError)

panda130 commented 3 years ago

Checklist

[x] I'm reporting a broken site support
[x] I've verified that I'm running youtube-dl version 2020.09.20
[x] I've checked that all provided URLs are alive and playable in a browser
[x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[x] I've searched the bugtracker for similar issues including closed ones

Verbose log

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-i', '-v', 'https://www.viu.com/ott/sg/en-us/vod/233440/Hot-Stove-League']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2020.09.20
[debug] Python version 3.7.7 (CPython) - Darwin-16.7.0-x86_64-i386-64bit
[debug] exe versions: ffmpeg 4.1.4, ffprobe 4.1.4, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {}
[viu:ott] 233440: Downloading video info
[viu:ott] 233440: Downloading stream info
ERROR: An extractor error has occurred. (caused by KeyError('data')); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 532, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/viu.py", line 233, in _real_extract
    })['data']['stream']
KeyError: 'data'
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 532, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/viu.py", line 233, in _real_extract
    })['data']['stream']
KeyError: 'data'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 797, in extract_info
    ie_result = ie.extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 545, in extract
    raise ExtractorError('An extractor error has occurred.', cause=e)
youtube_dl.utils.ExtractorError: An extractor error has occurred. (caused by KeyError('data')); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Description

As of today, the Viu OTT extractor is no longer working. I have tried multiple server locations.

This issue is occurring with videos that are geo-restricted to both Singapore and Thailand. They are still accessible via web browser using a proxy.

Singapore sample: https://www.viu.com/ott/sg/en-us/vod/233440/Hot-Stove-League Thailand sample: https://www.viu.com/ott/th/en-us/vod/165839/A-Poem-A-Day

This is really bad because Viu is one of the main sites for Asian TV dramas that are not available anywhere else. Thank you for your help!

zackmark29 commented 3 years ago

@dstftw

I also tested on my C# Tool. In PH, premium episode is now required premium account.

See this example:

FREE EPISODE: https://d1k2us671qcoau.cloudfront.net/distribute_web_ph.php?ccs_product_id=2b90508fc68055b6be8489381d5dd3a1&language_flag_id=3

PREMIUM EPISODE: https://d1k2us671qcoau.cloudfront.net/distribute_web_ph.php?ccs_product_id=353f52cfdf968f4cb195c9b7b29e4f69&language_flag_id=3&identity=8f36aeef79014287cbb770b75e688c4e1e1e8e20

panda130 commented 3 years ago

@zackmark29 Thank you for replying. I do not believe that either of the sample episodes I mentioned are premium episodes because you can watch them via web browser without a subscription.

Something seems to have changed to stop youtube-dl from even seeing the available video formats. Using "--list-formats" is not working.

So what you are saying is that even if youtube-dl could see the available video formats, the HD quality videos will no longer be available? The HD quality videos were working via youtube-dl a couple of days ago.

zackmark29 commented 3 years ago

@zackmark29 Thank you for replying. I do not believe that either of the sample episodes I mentioned are premium episodes because you can watch them via web browser without a subscription.

Something seems to have changed to stop youtube-dl from even seeing the available video formats. Using "--list-formats" is not working.

So what you are saying is that even if youtube-dl could see the available video formats, the HD quality videos will no longer be available? The HD quality videos were working via youtube-dl a couple of days ago.

No, in my case, if I don't have premium account, youtube-dl can't be able to check the available format as well. Because the cookie id is required.

I'll check now with SG vpn. I'll update you in a minute

zackmark29 commented 3 years ago

I'm able to get all resolution with you're given sg link

panda130 commented 3 years ago

@zackmark29 Thanks for following up. So it seems like all formats of the videos are still available. It's just that there is some syntax change on Viu's server side that the youtube-dl Viu extractor needs to compensate for.

zackmark29 commented 3 years ago

@zackmark29 Thanks for following up. So it seems like all formats of the videos are still available. It's just that there is some syntax change on Viu's server side that the youtube-dl Viu extractor needs to compensate for.

Yeah. If I just know Python I can edit it for you but I only know C# @dstftw can do it for you

Ashish0804 commented 3 years ago

I'm able to get all resolution with you're given sg link

Is your C# tool available to public?

zackmark29 commented 3 years ago

I'm able to get all resolution with you're given sg link

Is your C# tool available to public?

No, this is just for personal use.

Ashish0804 commented 3 years ago

I'm able to get all resolution with you're given sg link

Is your C# tool available to public?

No, this is just for personal use.

can u share the extractor code? I can try and modify it for python

zackmark29 commented 3 years ago

I'm able to get all resolution with you're given sg link

Is your C# tool available to public?

No, this is just for personal use.

can u share the extractor code? I can try and modify it for python

I just tested again and I noticed this is the problem Example url: https://www.viu.com/ott/sg/en-us/vod/233440/Hot-Stove-League

https://www.viu.com/ott/sg/index.php?r=vod/ajax-detail&platform_flag_label=web&area_id=2&language_flag_id=3&product_id=233440

to fix it should have &ut=2 at the end like this

https://www.viu.com/ott/sg/index.php?r=vod/ajax-detail&platform_flag_label=web&area_id=2&language_flag_id=3&product_id=233440&ut=2

And here's for the streams https://d1k2us671qcoau.cloudfront.net/distribute_web_sg.php?ccs_product_id=9aceb6645dbdd1afa252581c2476af8f&language_flag_id=3

for some premium videos sometimes token is required to get the streams.

https://d1k2us671qcoau.cloudfront.net/distribute_web_sg.php?ccs_product_id=9aceb6645dbdd1afa252581c2476af8f&language_flag_id=3&identity=6398e7db08dab6cdc285cdf26656723c31888c2e

Let me know if you needed more.

panda130 commented 3 years ago

@zackmark29 Yay! Thank you so much! Based on your observation, I fixed the viu.py extractor by adding in the 'language_flag_id' and the 'ut' into the query. I am not a programmer, and this is the first time I have ever done anything in Python...

I have also submitted my first-ever pull request.

zackmark29 commented 3 years ago

@zackmark29 Yay! Thank you so much! Based on your observation, I fixed the viu.py extractor by adding in the 'language_flag_id' and the 'ut' into the query. I am not a programmer, and this is the first time I have ever done anything in Python...

I have also submitted my first-ever pull request.

Wow that's great! so is it work now? I hope I also know how to modify python from youtube-dl extractor.

zackmark29 commented 3 years ago

But the ut i think it depends on the region. In my country (PH) it's ut=0 and for SG it's ut=2

panda130 commented 3 years ago

@zackmark29 I did some more testing. I actually removed the ut part of the query, and the extractor still works. So ut does not seem to be necessary.

There are a couple of remaining issues.

After I made the "language_flag_id" change to the query, the Viu extractor is only working for Singapore videos. For some reason, it does not work with Thailand videos. Can you find the correct video URL format for the Thailand videos? Sample URL: https://www.viu.com/ott/th/en-us/vod/165839/A-Poem-A-Day
The extractor is no longer working for Premium videos. It used to work for Premium videos.

zackmark29 commented 3 years ago

@zackmark29 I did some more testing. I actually removed the ut part of the query, and the extractor still works. So ut does not seem to be necessary.

There are a couple of remaining issues.

After I made the "language_flag_id" change to the query, the Viu extractor is only working for Singapore videos. For some reason, it does not work with Thailand videos. Can you find the correct video URL format for the Thailand videos? Sample URL: https://www.viu.com/ott/th/en-us/vod/165839/A-Poem-A-Day

The extractor is no longer working for Premium videos. It used to work for Premium videos.

In PH, ut is needed. But I'll look into it later.

touchhyuk commented 3 years ago

how can i download from viu with viu.py? i did install python but i can't run viu python. please guide

panda130 commented 3 years ago

@touchhyuk I assume that you've already downloaded the youtube-dl master source code. You'll need to make the modifications to viu.py that are listed in my pull request.

Then you have to recompile youtube-dl.

In your bash terminal, set your current directory to the root folder of the youtube-dl master source code. For instance, if you downloaded the youtube-dl source code to your Downloads folder, you might have to set your current directory to "/Users/USERNAMEHERE/Downloads/youtube-dl-master" by typing "cd /Users/USERNAMEHERE/Downloads/youtube-dl-master" in your bash terminal.

Then enter the command "make" in your bash terminal. There will be an executable binary file created called "youtube-dl". Move that "youtube-dl" binary file to wherever your computer normally stores the youtube-dl binary file. Use the terminal command "which youtube-dl" if you need to find out where that is.

touchhyuk commented 3 years ago

@touchhyuk I assume that you've already downloaded the youtube-dl master source code. You'll need to make the modifications to viu.py that are listed in my pull request.

Then you have to recompile youtube-dl.

In your bash terminal, set your current directory to the root folder of the youtube-dl master source code. For instance, if you downloaded the youtube-dl source code to your Downloads folder, you might have to set your current directory to "/Users/USERNAMEHERE/Downloads/youtube-dl-master" by typing "cd /Users/USERNAMEHERE/Downloads/youtube-dl-master" in your bash terminal.

Then enter the command "make" in your bash terminal. There will be an executable binary file created called "youtube-dl". Move that "youtube-dl" binary file to wherever your computer normally stores the youtube-dl binary file. Use the terminal command "which youtube-dl" if you need to find out where that is.

i cant run make command got this error

C:\Users\Administrator\Downloads\youtube-dl-master>make 'make' is not recognized as an internal or external command, operable program or batch file.

panda130 commented 3 years ago

@touchhyuk I am sorry to hear that. I don't use Windows so I'm afraid I can't help with this particular problem. It seems to be a permissions issue. Please try asking on the Reddit or stack exchange page.

See this link: https://stackoverflow.com/questions/55561387/how-to-rebuild-python-youtube-dl-from-the-source

lkho commented 3 years ago

@touchhyuk If you just want to run youtube-dl with python interpreter (which you have already downloaded), there is no need to recompile or rebuild a standalone exe. Just follow the developer instructions will work.

lkho commented 3 years ago

@panda130 @zackmark29 for your reference

In my country (HK), ut is also 0, area_id = 1, language_flag_id = 1 for zh-hk, 3 for en-us

touchhyuk commented 3 years ago

@touchhyuk If you just want to run youtube-dl with python interpreter (which you have already downloaded), there is no need to recompile or rebuild a standalone exe. Just follow the developer instructions will work.

i cant please giive your discord id or telegram id

touchhyuk commented 3 years ago

please someone fix youtube-dl.exe with viu.py and send here. i want downloaf from viu and tried a week but cant fix

zackmark29 commented 3 years ago

please someone fix youtube-dl.exe with viu.py and send here. i want downloaf from viu and tried a week but cant fix

You could try to follow my instruction from this https://github.com/ytdl-org/youtube-dl/issues/26788

ytdl-org / youtube-dl