edsu / earls

display urls being tweeted with an event hashtag
MIT License
18 stars 3 forks source link

"403 Forbidden" when it's scrapping Buffer App's shorturl #10

Closed remagio closed 9 years ago

remagio commented 9 years ago

It happens with earls.js and load.js, results are:

imho I think Buffer is blocking only when it try to scrape the document's title

All urls get same description inside earls log, including when happen quotes/retweets:

processing tweet: 573110409865105408
queueing lookup for http://buff.ly/1F8MJYW
looking up url: http://buff.ly/1F8MJYW
tallying:  http://gilda35.com/2013/12/anche-gli-androidi-votano-i-candidati-su-twitter.html?utm_content=buffer45750&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer#axzz3TPDNsgND 403 Forbidden Data88Smart https://twitter.com/Data88Smart/statuses/573110409865105408
finished processing url: http://buff.ly/1F8MJYW
edsu commented 9 years ago

Seems to work now after the HTTP request has a common User-Agent:

processing tweet: 573106948922609664
looking up url: http://buff.ly/1F8MJYW
queueing lookup for http://buff.ly/1F8MJYW
tallying:  http://gilda35.com/2013/12/anche-gli-androidi-votano-i-candidati-su-twitter.html?utm_content=buffer45750&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer#axzz3TPDNsgND Anche gli androidi votano i candidati su twitter - Gilda35, satira dadaista sul professionismo di internet Gilda35 https://twitter.com/Gilda35/statuses/573106948922609664