search

ushahidi / SwiftRiver-Core

SwiftRiver Core Applications
6 stars 3 forks source link

Improve short url expansion #11

Closed 69mb closed 12 years ago

69mb commented 12 years ago

The current method relies on httplib2 doing HEAD request and following any redirects that follow on any url that looks shortened. This method will however freeze if the destination url is a streaming URL and there is a 302 http response causing the HEAD request to be changed into a GET request.

Probably only expand urls from known services only or find a clean way to prevent the GET request...

import httplib2 httplib2.debuglevel=10 h = httplib2.Http() resp, content = h.request("http://t.co/vO2H0QLk", "HEAD") connect: (t.co, 80) send: 'HEAD /vO2H0QLk HTTP/1.1\r\nHost: t.co\r\naccept-encoding: gzip, deflate\r\nuser-agent: Python-httplib2/0.7.4 (gzip)\r\n\r\n' reply: 'HTTP/1.1 301 Moved Permanently\r\n' header: Date: Sat, 02 Jun 2012 16:54:57 GMT header: Server: hi header: Location: http://usa1-pn.mixstream.net/8006.m header: Cache-Control: private,max-age=300 header: Expires: Sat, 02 Jun 2012 16:59:57 GMT header: Connection: close header: Content-Type: text/html; charset=UTF-8 connect: (usa1-pn.mixstream.net, 80) send: 'HEAD /8006.m HTTP/1.1\r\nHost: usa1-pn.mixstream.net\r\naccept-encoding: gzip, deflate\r\nuser-agent: Python-httplib2/0.7.4 (gzip)\r\n\r\n' reply: 'HTTP/1.1 302 Found\r\n' header: Date: Sat, 02 Jun 2012 16:54:36 GMT header: Server: Apache/2.0.52 (CentOS) header: X-Powered-By: PHP/4.3.9 header: Location: http://usa1-pn.mixstream.net:8006/;listen.mp3 header: Connection: close header: Content-Type: text/html; charset=UTF-8 connect: (usa1-pn.mixstream.net, 8006) send: 'GET /;listen.mp3 HTTP/1.1\r\nHost: usa1-pn.mixstream.net:8006\r\naccept-encoding: gzip, deflate\r\nuser-agent: Python-httplib2/0.7.4 (gzip)\r\n\r\n' reply: 'ICY 200 OK\r\n'