search

bvanheu / pytoutv

TOU.TV client library and user interfaces written in Python 3
96 stars 23 forks source link

fetch: too many API calls #104

Closed gboudreau closed 7 years ago

gboudreau commented 7 years ago

It looks like we do too many API calls when trying to download episodes, especially when logged in.

There's a few API calls that are exactly the same that I'm pretty sure we could skip, while others are very similar, and I'll try to validate if they can be skipped.

Verbose log:

DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): services.radio-canada.ca
DEBUG:requests.packages.urllib3.connectionpool:https://services.radio-canada.ca:443 "GET /media/validation/v2/GetClaims?token=... HTTP/1.1" 200 388
DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): services.radio-canada.ca
DEBUG:requests.packages.urllib3.connectionpool:https://services.radio-canada.ca:443 "GET /media/validation/v2/?deviceType=iphone4&idMedia=126337&output=json&appCode=toutv&connectionType=wifi&claims=... HTTP/1.1" 200 400
DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): toutvuniver1-vh.akamaihd.net
DEBUG:requests.packages.urllib3.connectionpool:https://toutvuniver1-vh.akamaihd.net:443 "GET /i/012/mp4/l/2016-12-02_19_30_00_lepharmachien_0001_,500,800,1200,2000,3000,.mp4.csmil/master.m3u8?hdnea=st=1484595565~exp=1484595630~acl=/i/012/mp4/l/2016-12-02_19_30_00_lepharmachien_0001_*~hmac=... HTTP/1.1" 200 2637
DEBUG:Downloader:starting download
DEBUG:ToutvApiSegmentProvider:episode: Épisode 01 (2425169943)
DEBUG:ToutvApiSegmentProvider:bitrate: 3126000
DEBUG:ToutvApiSegmentProvider:timeout: 15
DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): services.radio-canada.ca
DEBUG:requests.packages.urllib3.connectionpool:https://services.radio-canada.ca:443 "GET /media/validation/v2/GetClaims?token=... HTTP/1.1" 200 388
DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): services.radio-canada.ca
DEBUG:requests.packages.urllib3.connectionpool:https://services.radio-canada.ca:443 "GET /media/validation/v2/?deviceType=iphone4&idMedia=126337&output=json&appCode=toutv&connectionType=wifi&claims=... HTTP/1.1" 200 400
DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): toutvuniver1-vh.akamaihd.net
DEBUG:requests.packages.urllib3.connectionpool:https://toutvuniver1-vh.akamaihd.net:443 "GET /i/012/mp4/l/2016-12-02_19_30_00_lepharmachien_0001_,500,800,1200,2000,3000,.mp4.csmil/master.m3u8?hdnea=st=1484595566~exp=1484595631~acl=/i/012/mp4/l/2016-12-02_19_30_00_lepharmachien_0001_*~hmac=... HTTP/1.1" 200 2647
DEBUG:ToutvApiSegmentProvider:HTTP GET request @ https://toutvuniver1-vh.akamaihd.net/i/012/mp4/l/2016-12-02_19_30_00_lepharmachien_0001_,500,800,1200,2000,3000,.mp4.csmil/index_4_av.m3u8?null=0&id=AgBoo5m0YH%2f6ZHEhfVhfJgzbdVfMTZmZFUEl5H2%2fN5HEyQCggSV3rFKviROLp4O5QfE6+uBJVWZgEA%3d%3d&hdntl=exp=1484681969~acl=%2fi%2f012%2fmp4%2fl%2f2016-12-02_19_30_00_lepharmachien_0001_*~data=hdntl~hmac=...
DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): toutvuniver1-vh.akamaihd.net
DEBUG:requests.packages.urllib3.connectionpool:https://toutvuniver1-vh.akamaihd.net:443 "GET /i/012/mp4/l/2016-12-02_19_30_00_lepharmachien_0001_,500,800,1200,2000,3000,.mp4.csmil/index_4_av.m3u8?null=0&id=AgBoo5m0YH%2f6ZHEhfVhfJgzbdVfMTZmZFUEl5H2%2fN5HEyQCggSV3rFKviROLp4O5QfE6+uBJVWZgEA%3d%3d&hdntl=exp=1484681969~acl=%2fi%2f012%2fmp4%2fl%2f2016-12-02_19_30_00_lepharmachien_0001_*~data=hdntl~hmac=... HTTP/1.1" 200 59666
DEBUG:ToutvApiSegmentProvider:parsed M3U8 file: 142 total segments
DEBUG:ToutvApiSegmentProvider:HTTP GET request @ https://toutvuniver1-vh.akamaihd.net/i/012/mp4/l/2016-12-02_19_30_00_lepharmachien_0001_,500,800,1200,2000,3000,.mp4.csmil/crypt.key?null=0&id=AgBoo5m0YH%2f6ZHEhfVhfJgzbdVfMTZmZFUEl5H2%2fN5HEyQCggSV3rFKviROLp4O5QfE6+uBJVWZgEA%3d%3d&hdntl=exp=1484681969~acl=/i/012/mp4/l/2016-12-02_19_30_00_lepharmachien_0001_*~data=hdntl~hmac=...
DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): toutvuniver1-vh.akamaihd.net
DEBUG:requests.packages.urllib3.connectionpool:https://toutvuniver1-vh.akamaihd.net:443 "GET /i/012/mp4/l/2016-12-02_19_30_00_lepharmachien_0001_,500,800,1200,2000,3000,.mp4.csmil/crypt.key?null=0&id=AgBoo5m0YH%2f6ZHEhfVhfJgzbdVfMTZmZFUEl5H2%2fN5HEyQCggSV3rFKviROLp4O5QfE6+uBJVWZgEA%3d%3d&hdntl=exp=1484681969~acl=/i/012/mp4/l/2016-12-02_19_30_00_lepharmachien_0001_*~data=hdntl~hmac=,,, HTTP/1.1" 200 16
DEBUG:ToutvApiSegmentProvider:decryption key: ...
DEBUG:FilesystemSegmentHandler:episode: Épisode 01 (2425169943)
DEBUG:FilesystemSegmentHandler:bitrate: 3126000
DEBUG:FilesystemSegmentHandler:output path: Les.aventures.du.pharmachien.S01E01.Épisode.01.3126kbps.ts
DEBUG:FilesystemSegmentHandler:overwrite: False
simark commented 7 years ago

The claims queries seems to be duplicated:

      2 DEBUG:requests.packages.urllib3.connectionpool:https://services.radio-canada.ca:443 "GET /media/validation/v2/?deviceType=iphone4&idMedia=126337&output=json&appCode=toutv&connectionType=wifi&claims=... HTTP/1.1" 200 400
      2 DEBUG:requests.packages.urllib3.connectionpool:https://services.radio-canada.ca:443 "GET /media/validation/v2/GetClaims?token=... HTTP/1.1" 200 388

It looks like they are fetched for every request by the Auth object (in auth.py). I suppose that we could save the result of the first request for the other ones?

gboudreau commented 7 years ago

Yes, did that. And same for playlist and cookies in Episode. And will try to check for the local file existence before the other two API calls; I doubt that is needed...

gboudreau commented 7 years ago

End result:

DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): services.radio-canada.ca
DEBUG:requests.packages.urllib3.connectionpool:https://services.radio-canada.ca:443 "GET /media/validation/v2/GetClaims?token=d3a90209-8d0a-449d-b920-6573cf8a90cc HTTP/1.1" 200 388
DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): services.radio-canada.ca
DEBUG:requests.packages.urllib3.connectionpool:https://services.radio-canada.ca:443 "GET /media/validation/v2/?deviceType=iphone4&appCode=toutv&output=json&claims=FF991AC02DA2277DD55C46CECC7BD2D8A51A9B3845611C3F229F6C5D4AF2B755A7C245A4EB9AD090F56CB9A765EAD68AAC638751CE90500AC51F09445E90029918637AB59692D0B3D292ABDDC09A8AD698366AC306585D42152350C03A2AE7E8AB61E41F1A6CFE65359E91A8568F4C58C7BF3F0DF020919149AAC6F190971078&idMedia=126970&connectionType=wifi HTTP/1.1" 200 392
DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): toutvuniver1-vh.akamaihd.net
DEBUG:requests.packages.urllib3.connectionpool:https://toutvuniver1-vh.akamaihd.net:443 "GET /i/012/mp4/r/2017-01-07_07_59_40_rangerrob_0003_,500,800,1200,2000,3000,.mp4.csmil/master.m3u8?hdnea=st=1484597202~exp=1484597267~acl=/i/012/mp4/r/2017-01-07_07_59_40_rangerrob_0003_*~hmac=1d55416a8bc16c316085d5370a227dab03733e37542cf8a844320b37bfcf2862 HTTP/1.1" 200 2587
DEBUG:Downloader:starting download
DEBUG:FilesystemSegmentHandler:episode: Épisode 03 (2425170576)
DEBUG:FilesystemSegmentHandler:bitrate: 561000
DEBUG:FilesystemSegmentHandler:output path: /Users/gb/git/pytoutv/toutvcli/Ranger.Rob.S01E03.Épisode.03.561kbps.ts
DEBUG:FilesystemSegmentHandler:overwrite: False

And the Claims API call will not be repeated until the app is restarted, so trying to download multiple episodes will only call that API endpoint once.

gboudreau commented 7 years ago

What was previously 405 API calls to try to see if any new episodes of a few different shows were now available is now 100 API calls! Hooray.

simark commented 7 years ago

Wow, that's significant! Depending on the number of episodes, 100 still sounds quite a lot.

gboudreau commented 7 years ago

It's two API calls per episode: one to get the playlist URL and cookies, and the other one to load that playlist, to get the available qualities. Plus three API calls only executed on start, to load all shows, the specified show, and get the claims.