Bad HTTP status code (404) for "https://ici.tou.tv/presentation/search" (was L'error se produit en faisant l'appel à l'API toutv)

[sirber]

May be not related to pytoutv but...

toutv fetch -q MAX "UNITE-9" S07E02 Unknown exception: <class 'RuntimeError'>: L'error se produit en faisant l'appel à l'API toutv

[simark]

Full backtrace, for reference:

$ toutv -n -v fetch -q MAX "UNITE-9" S07E02
_parse_show_episode_from_args returns: emission UNITE-9 and episode S07E02
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): ici.tou.tv
DEBUG:urllib3.connectionpool:https://ici.tou.tv:443 "GET /presentation/search?v=2&d=android HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): ici.tou.tv
DEBUG:urllib3.connectionpool:https://ici.tou.tv:443 "GET /presentation//unite-9?v=2&excludeLineups=False&d=android HTTP/1.1" 200 38490
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): api.radio-canada.ca
DEBUG:urllib3.connectionpool:http://api.radio-canada.ca:80 "GET /validationMedia/v1/Validation.html?connectionType=wifi&deviceType=iphone4&appCode=toutv&idMedia=140892&output=json HTTP/1.1" 200 137
Unknown exception: <class 'RuntimeError'>: L'error se produit en faisant l'appel à l'API toutv
Traceback (most recent call last):
  File "/home/emaisin/src/pytoutv/toutvcli/app.py", line 110, in run
    args.func(args)
  File "/home/emaisin/src/pytoutv/toutvcli/app.py", line 506, in _command_fetch
    self._fetch_episode(episode, output_dir=output_dir, quality=quality, bitrate=bitrate, overwrite=overwrite)
  File "/home/emaisin/src/pytoutv/toutvcli/app.py", line 733, in _fetch_episode
    qualities = episode.get_available_qualities()
  File "/home/emaisin/src/pytoutv/toutv/bos.py", line 554, in get_available_qualities
    playlist, cookies = self.get_playlist_cookies()
  File "/home/emaisin/src/pytoutv/toutv/bos.py", line 542, in get_playlist_cookies
    url = self._get_playlist_url()
  File "/home/emaisin/src/pytoutv/toutv/bos.py", line 532, in _get_playlist_url
    raise RuntimeError(response_obj['message'])
RuntimeError: L'error se produit en faisant l'appel à l'API toutv

[simark]

I get this with any show/episode. Time to get mitm out I suppose!

[sirber]

from simark's debug:

http://api.radio-canada.ca/validationMedia/v1/Validation.html?connectionType=wifi&deviceType=iphone4&appCode=toutv&idMedia=140892&output=json

{"url":null,"message":"L\u0027error se produit en faisant l\u0027appel à l\u0027API toutv","errorCode":37,"params":null,"bitrates":null}

[simark]

Also, youtube-dl has the exact same issue:

youtube-dl -v https://ici.tou.tv/l-epicerie/S17E02?lectureauto=1     
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'https://ici.tou.tv/l-epicerie/S17E02?lectureauto=1']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.09.18
[debug] Python version 3.5.2 (CPython) - Linux-4.15.0-15-generic-x86_64-with-Ubuntu-16.04-xenial
[debug] exe versions: ffmpeg 2.8.15-0ubuntu0.16.04.1, ffprobe 2.8.15-0ubuntu0.16.04.1
[debug] Proxy map: {}
[tou.tv] l-epicerie/S17E02: Downloading JSON metadata
[radiocanada] 141087: Downloading metadata XML
[radiocanada] 141087: Downloading ipad XML
[radiocanada] 141087: Downloading flash XML
[radiocanada] 141087: Downloading android XML
ERROR: radiocanada said: L'error se produit en faisant l'appel à l'API toutv
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/youtube_dl/YoutubeDL.py", line 792, in extract_info
    ie_result = ie.extract(url)
  File "/usr/local/lib/python3.5/dist-packages/youtube_dl/extractor/common.py", line 507, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/lib/python3.5/dist-packages/youtube_dl/extractor/radiocanada.py", line 152, in _real_extract
    '%s said: %s' % (self.IE_NAME, error), expected=True)
youtube_dl.utils.ExtractorError: radiocanada said: L'error se produit en faisant l'appel à l'API toutv

[simark]

Ok, using today's Android versions, it's not as easy to make the tou.tv app trust a third-party certificate, a little more work is necessary:

https://www.nccgroup.trust/uk/about-us/newsroom-and-events/blogs/2017/november/bypassing-androids-network-security-configuration/

[sirber]

you want to proxy the android app to a proxy you control to grab the video?

[simark]

I indeed want to proxy the Android app to a proxy I control (mitmproxy), but to understand how the application fetches the video, so we can replicate it in pytoutv. It seems like tou.tv uses a completely different system than before, based on Microsoft Smooth Streaming.

[sirber]

Did they deprecate their v1 api?

[simark]

Did they deprecate their v1 api?

I don't know, which one is that? The one the current pytoutv code is using or the one we used a few years ago?

[sirber]

From my understanding pytoutv uses the api v1, as seen in the url. I don;t know how it was working earler. I'm sure you know much more than me on this lol.

[simark]

From my understanding pytoutv uses the api v1, as seen in the url.

Oh right. There are a bunch of v2 urls in config.py, but that's for authentication, so probably a different service.

[simark]

So I don't know whether the v1 API is deprecated. We would need to find a device/app that uses it that still works an inspect its traffic. I would be curious to see a traffic dump from an iDevice.

[simark]

Hmm it seems to work here now... can you check? Maybe it was just a small outage of their API...

[sirber]

Bad HTTP status code (404) for "https://ici.tou.tv/presentation/search"

[sirber]

works!!