search

bvanheu / pytoutv

TOU.TV client library and user interfaces written in Python 3
96 stars 23 forks source link

"certificate verify failed" when downloading #47

Closed anarcat closed 9 years ago

anarcat commented 9 years ago

there seems to be certain videos that fail to download because of problems with HTTPS:

$ toutv fetch Infoman S15E16
Unknown error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)

wget seems to be able to get the file correctly, however:

$ strace -s 8192 -e recvfrom toutv fetch Infoman S15E16
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=24340, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=24341, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
recvfrom(5, "\7\216\201\200\0\1\0\2\0\1\0\0\3api\fradio-canada\2ca\0\0\34\0\1\300\f\0\5\0\1\0\0\7\324\0\34\3san\7src-cbc\2ca\7edgekey\3net\0\3001\0\5\0\1\0\0>\334\0\25\5e5627\1g\nakamaiedge\300H\300_\0\6\0\1\0\0\2M\0001\3n0g\300a\nhostmaster\6akamai\3com\0T\253\\e\0\0\3\350\0\0\3\350\0\0\3\350\0\0\7\10", 2048, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.0.1")}, [16]) = 171
recvfrom(5, "|\342\201\200\0\1\0\3\0\0\0\0\3api\fradio-canada\2ca\0\0\1\0\1\300\f\0\5\0\1\0\0\7\324\0\34\3san\7src-cbc\2ca\7edgekey\3net\0\3001\0\5\0\1\0\0>\334\0\25\5e5627\1g\nakamaiedge\300H\300Y\0\1\0\1\0\0\0\24\0\4\27\t`\311", 1877, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.0.1")}, [16]) = 126
recvfrom(5, "HTTP/1.1 200 OK\r\nServer: Microsoft-IIS/7.5\r\nContent-Length: 382\r\nContent-Type: application/json\r\nAccess-Control-Allow-Headers: *\r\nAccess-Control-Allow-Origin: *\r\nX-Powered-By: ASP.NET\r\nETag: \"90aeefc1affecf1:0\"\r\nDate: Tue, 06 Jan 2015 04:01:05 GMT\r\nConnection: keep-alive\r\n\r\n{\"url\":\"https://toutvuniver1-vh.akamaihd.net/i/012/mp4/i/2014-12-31_22_00_00_infoman_0385_,400,500,800,1200,.mp4.csmil/master.m3u8?hdnea=st=1420516862~exp=1420516877~acl=/i/012/mp4/i/2014-12-31_22_00_00_infoman_0385_*~hmac=b7beecbc39e2c89fc033f8946e74d0771adb92acbbe36d75332ace8c4722d5ed\",\"message\":null,\"errorCode\":0,\"params\":[{\"name\":\"mediaType\",\"value\":\"video\"}],\"bitrates\":null}", 8192, 0, NULL, NULL) = 657
recvfrom(5, "\360-\201\200\0\1\0\1\0\1\0\0\17toutvuniver1-vh\10akamaihd\3net\0\0\34\0\1\300\f\0\5\0\1\0\0\0\362\0\22\4a214\3w10\6akamai\300%\300?\0\6\0\1\0\0\2N\0003\5n0w10\300C\nhostmaster\6akamai\3com\0T\253\\f\0\0\3\350\0\0\3\350\0\0\3\350\0\0\7\10", 2048, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.0.1")}, [16]) = 139
recvfrom(5, "wI\201\200\0\1\0\3\0\0\0\0\17toutvuniver1-vh\10akamaihd\3net\0\0\1\0\1\300\f\0\5\0\1\0\0\0\362\0\22\4a214\3w10\6akamai\300%\300:\0\1\0\1\0\0\0\24\0\4\330\234\307*\300:\0\1\0\1\0\0\0\24\0\4\330\234\307)", 1909, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.0.1")}, [16]) = 108
Unknown error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)
+++ exited with 100 +++
$ wget https://toutvuniver1-vh.akamaihd.net/i/012/mp4/i/2014-12-31_22_00_00_infoman_0385_,400,500,800,1200,.mp4.csmil/master.m3u8?hdnea=st=1420516862~exp=1420516877~acl=/i/012/mp4/i/2014-12-31_22_00_00_infoman_0385_*~hmac=b7beecbc39e2c89fc033f8946e74d0771adb92acbbe36d75332ace8c4722d5ed
--2015-01-05 23:01:15--  https://toutvuniver1-vh.akamaihd.net/i/012/mp4/i/2014-12-31_22_00_00_infoman_0385_,400,500,800,1200,.mp4.csmil/master.m3u8?hdnea=st=1420516862~exp=1420516877~acl=/i/012/mp4/i/2014-12-31_22_00_00_infoman_0385_*~hmac=b7beecbc39e2c89fc033f8946e74d0771adb92acbbe36d75332ace8c4722d5ed
Résolution de toutvuniver1-vh.akamaihd.net (toutvuniver1-vh.akamaihd.net)… 216.156.199.42, 216.156.199.41
Connexion à toutvuniver1-vh.akamaihd.net (toutvuniver1-vh.akamaihd.net)|216.156.199.42|:443… connecté.
requête HTTP transmise, en attente de la réponse… 200 OK
Taille : 1285 (1,3K) [application/vnd.apple.mpegurl]
Sauvegarde en : « master.m3u8?hdnea=st=1420516862~exp=1420516877~acl=%2Fi%2F012%2Fmp4%2Fi%2F2014-12-31_22_00_00_infoman_0385_*~hmac=b7beecbc39e2c89fc033f8946e74d0771adb92acbbe36d75332ace8c4722d5ed »

master.m3u8?hdnea=st=1420516862~exp=1420516877~acl=%2Fi%2F0 100%[============================================================================================================================================>]   1,25K  --.-KB/s   ds 0s

2015-01-05 23:01:15 (13,1 MB/s) — « master.m3u8?hdnea=st=1420516862~exp=1420516877~acl=%2Fi%2F012%2Fmp4%2Fi%2F2014-12-31_22_00_00_infoman_0385_*~hmac=b7beecbc39e2c89fc033f8946e74d0771adb92acbbe36d75332ace8c4722d5ed » sauvegardé [1285/1285]

Now, interestingly, other episodes seem to download fine:

$ toutv fetch -f Enquête S2014E11
Enquête.S2014E11.La.guerre.des.bois.925kbps.ts                                                                                    776.0 kiB     0/260 [----------------------------------------------------------------------------------]   0%

... this one is not served from an HTTPS url!

$ strace -e recvfrom -s 4096 toutv fetch -f Enquête S2014E11
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=24760, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=24761, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
recvfrom(5, "s\351\201\200\0\1\0\3\0\0\0\0\3api\fradio-canada\2ca\0\0\1\0\1\300\f\0\5\0\1\0\0\7M\0\34\3san\7src-cbc\2ca\7edgekey\3net\0\3001\0\5\0\1\0\0>U\0\25\5e5627\1g\nakamaiedge\300H\300Y\0\1\0\1\0\0\0\5\0\4\27\t`\311", 2048, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.0.1")}, [16]) = 126
recvfrom(5, "V\234\201\200\0\1\0\2\0\1\0\0\3api\fradio-canada\2ca\0\0\34\0\1\300\f\0\5\0\1\0\0\7M\0\34\3san\7src-cbc\2ca\7edgekey\3net\0\3001\0\5\0\1\0\0>U\0\25\5e5627\1g\nakamaiedge\300H\300_\0\6\0\1\0\0\1\306\0001\3n0g\300a\nhostmaster\6akamai\3com\0T\253\\e\0\0\3\350\0\0\3\350\0\0\3\350\0\0\7\10", 1922, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.0.1")}, [16]) = 171
recvfrom(5, "HTTP/1.1 200 OK\r\nServer: Microsoft-IIS/7.5\r\nContent-Length: 368\r\nContent-Type: application/json\r\nAccess-Control-Allow-Headers: *\r\nAccess-Control-Allow-Origin: *\r\nX-Powered-By: ASP.NET\r\nETag: \"90aeefc1affecf1:0\"\r\nDate: Tue, 06 Jan 2015 04:03:19 GMT\r\nConnection: keep-alive\r\n\r\n{\"url\":\"http://cp143903-f.akamaihd.net/i/012/mp4/e/2014-12-04_21_00_00_enq_0174_,400,500,800,1200,.mp4.csmil/master.m3u8?hdnea=st=1420516996~exp=1420517011~acl=/i/012/mp4/e/2014-12-04_21_00_00_enq_0174_*~hmac=595a0a471796cda7f848bdc72d6e6964faa157c1b43ad67066fce35949fb5038\",\"message\":null,\"errorCode\":0,\"params\":[{\"name\":\"mediaType\",\"value\":\"video\"}],\"bitrates\":null}", 8192, 0, NULL, NULL) = 643
recvfrom(5, "\3755\201\200\0\1\0\3\0\0\0\0\ncp143903-f\10akamaihd\3net\0\0\1\0\1\300\f\0\5\0\1\0\0\0|\0\20\2a5\3w23\6akamai\300 \3005\0\1\0\1\0\0\0\5\0\4\314\234\0170\3005\0\1\0\1\0\0\0\5\0\4\314\234\17\t", 2048, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.0.1")}, [16]) = 101
recvfrom(5, "9\372\201\200\0\1\0\1\0\1\0\0\ncp143903-f\10akamaihd\3net\0\0\34\0\1\300\f\0\5\0\1\0\0\0|\0\20\2a5\3w23\6akamai\300 \3008\0\6\0\1\0\0\2\7\0003\5n0w23\300<\nhostmaster\6akamai\3com\0T\253\\\247\0\0\3\350\0\0\3\350\0\0\3\350\0\0\7\10", 1947, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.0.1")}, [16]) = 132
recvfrom(5, "HTTP/1.0 200 OK\r\nServer: AkamaiGHost\r\nMime-Version: 1.0\r\nContent-Type: application/vnd.apple.mpegurl\r\nContent-Length: 1551\r\nPragma: no-cache\r\nCache-Control: no-store\r\nExpires: Tue, 06 Jan 2015 04:03:19 GMT\r\nDate: Tue, 06 Jan 2015 04:03:19 GMT\r\nConnection: keep-alive\r\nSet-Cookie: hdntl=exp=1420603399~acl=%2fi%2f012%2fmp4%2fe%2f2014-12-04_21_00_00_enq_0174_*~data=hdntl~hmac=351f90dca30e149492e8311aa6fb071c6603f636497b0d5eaa8841ebadca2eb4; path=/; domain=cp143903-f.akamaihd.net; \r\nSet-Cookie: _alid_=OgZcl4MfcHGlfU/wKcdgUg==; path=/i//012/mp4/e/2014-12-04_21_00_00_enq_0174_,400,500,800,1200,.mp4.csmil/; domain=cp143903-f.akamaihd.net\r\n\r\n#EXTM3U\n#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=461000,RESOLUTION=480x270,CODECS=\"avc1.66.30, mp4a.40.2\"\nhttp://cp143903-f.akamaihd.net/i/012/mp4/e/2014-12-04_21_00_00_enq_0174_,400,500,800,1200,.mp4.csmil/index_0_av.m3u8?null=&id=AgBIAEiQzJwPLIdeq1TtNog05%2fxoLFnr7XwBzkt%2fownbcelxeg8u2bC831MJyKDEMpgL+HRj%2fQnw1A%3d%3d\n#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=561000,RESOLUTION=512x288,CODECS=\"avc1.66.30, mp4a.40.2\"\nhttp://cp143903-f.akamaihd.net/i/012/mp4/e/2014-12-04_21_00_00_enq_0174_,400,500,800,1200,.mp4.csmil/index_1_av.m3u8?null=&id=AgBIAEiQzJwPLIdeq1TtNog05%2fxoLFnr7XwBzkt%2fownbcelxeg8u2bC831MJyKDEMpgL+HRj%2fQnw1A%3d%3d\n#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=925000,RESOLUTION=", 8192, 0, NULL, NULL) = 1338
recvfrom(5, "640x360,CODECS=\"avc1.66.30, mp4a.40.2\"\nhttp://cp143903-f.akamaihd.net/i/012/mp4/e/2014-12-04_21_00_00_enq_0174_,400,500,800,1200,.mp4.csmil/index_2_av.m3u8?null=&id=AgBIAEiQzJwPLIdeq1TtNog05%2fxoLFnr7XwBzkt%2fownbcelxeg8u2bC831MJyKDEMpgL+HRj%2fQnw1A%3d%3d\n#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1325000,RESOLUTION=852x480,CODECS=\"avc1.77.30, mp4a.40.2\"\nhttp://cp143903-f.akamaihd.net/i/012/mp4/e/2014-12-04_21_00_00_enq_0174_,400,500,800,1200,.mp4.csmil/index_3_av.m3u8?null=&id=AgBIAEiQzJwPLIdeq1TtNog05%2fxoLFnr7XwBzkt%2fownbcelxeg8u2bC831MJyKDEMpgL+HRj%2fQnw1A%3d%3d\n#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=61000,CODECS=\"mp4a.40.2\"\nhttp://cp143903-f.akamaihd.net/i/012/mp4/e/2014-12-04_21_00_00_enq_0174_,400,500,800,1200,.mp4.csmil/index_0_a.m3u8?null=&id=AgBIAEiQzJwPLIdeq1TtNog05%2fxoLFnr7XwBzkt%2fownbcelxeg8u2bC831MJyKDEMpgL+HRj%2fQnw1A%3d%3d\n", 8192, 0, NULL, NULL) = 854
recvfrom(5, "\324~\201\200\0\1\0\3\0\0\0\0\3api\fradio-canada\2ca\0\0\1\0\1\300\f\0\5\0\1\0\0\7M\0\34\3san\7src-cbc\2ca\7edgekey\3net\0\3001\0\5\0\1\0\0>U\0\25\5e5627\1g\nakamaiedge\300H\300Y\0\1\0\1\0\0\0\5\0\4\27\t`\311", 2048, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.0.1")}, [16]) = 126
recvfrom(5, "@\32\201\200\0\1\0\2\0\1\0\0\3api\fradio-canada\2ca\0\0\34\0\1\300\f\0\5\0\1\0\0\7M\0\34\3san\7src-cbc\2ca\7edgekey\3net\0\3001\0\5\0\1\0\0>U\0\25\5e5627\1g\nakamaiedge\300H\300_\0\6\0\1\0\0\1\306\0001\3n0g\300a\nhostmaster\6akamai\3com\0T\253\\e\0\0\3\350\0\0\3\350\0\0\3\350\0\0\7\10", 1922, 0, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.0.1")}, [16]) = 171
recvfrom(5, "HTTP/1.1 200 OK\r\nServer: Microsoft-IIS/7.5\r\nContent-Length: 368\r\nContent-Type: application/json\r\nAccess-Control-Allow-Headers: *\r\nAccess-Control-Allow-Origin: *\r\nX-Powered-By: ASP.NET\r\nETag: \"90aeefc1affecf1:0\"\r\nDate: Tue, 06 Jan 2015 04:03:20 GMT\r\nConnection: keep-alive\r\n\r\n{\"url\":\"http://cp143903-f.akamaihd.net/i/012/mp4/e/2014-12-04_21_00_00_enq_0174_,400,500,800,1200,.mp4.csmil/master.m3u8?hdnea=st=1420516997~exp=1420517012~acl=/i/012/mp4/e/2014-12-04_21_00_00_enq_0174_*~hmac=2876c31efe9ef9413854fd4791a5753a1c1e70f0086583dd537ab220e90aa8d0

so i am guessing the HTTPS validation methods are failing here.

simark commented 9 years ago

I'll just mention that I can't reproduce it here, so I won't be able to help. "toutv fetch Infoman S15E16" downloads fine. I checked with strace like you do, it indeed downloads from HTTPS (https://toutvuniver1-vh.akamaihd.net/...).

anarcat commented 9 years ago

i can confirm this is a problem with the "requests" package, at least in Debian Jessie:

ii  python3-requests        2.4.3-4          all              elegant and simple HTTP library for Python3, built f
ii  python3                 3.4.2-2          amd64            interactive high-level object-oriented language (def
$ python3 -c "import requests; r=requests.get('https://toutvuniver1-vh.akamaihd.net/i/012/mp4/i/2014-12-31_22_00_00_infoman_0385_,400,500,800,1200,.mp4.csmil/master.m3u8?hdnea=st=1420516862~exp=1420516877~acl=/i/012/mp4/i/2014-12-31_22_00_00_infoman_0385_*~hmac=b7beecbc39e2c89fc033f8946e74d0771adb92acbbe36d75332ace8c4722d5ed'); print(r.status_code)"
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 516, in urlopen
    body=body, headers=headers)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 304, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 724, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 237, in connect
    ssl_version=resolved_ssl_version)
  File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 123, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3.4/ssl.py", line 364, in wrap_socket
    _context=self)
  File "/usr/lib/python3.4/ssl.py", line 577, in __init__
    self.do_handshake()
  File "/usr/lib/python3.4/ssl.py", line 804, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 362, in send
    timeout=timeout
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 543, in urlopen
    raise SSLError(e)
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/requests/api.py", line 60, in get
    return request('get', url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 49, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 457, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 569, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 420, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)

the server's HTTPS configuration is less than desirable but that shouldn't stop us from downloading funny videos. :)

anarcat commented 9 years ago

oh wow, totally weird: it turns out i had one certificate disabled in the ca-certificates package, and that turns out to be the one used by akamai. bloody hell... sorry for the noise, dpkg-reconfigure ca-certificates fixed it. :)

simark commented 9 years ago

Phew!