search

VikParuchuri / apartment-finder

A Slack bot that helps you find an apartment.
MIT License
1.07k stars 325 forks source link

international craigslist URL #7

Open jonpolak opened 8 years ago

jonpolak commented 8 years ago

for cities outside the US craigslist redirects away from the .org domain. e.g. vancouver.craigslist.ca - any way to configure this?

WIRUT commented 8 years ago

@jonpolak I was able to get it to work outside of the US. Without knowing what you have as your arguments, can't really see where you're going wrong.

jonpolak commented 8 years ago

what were the settings you used in your test? @WIRUT

jonpolak commented 8 years ago

@WIRUT are you using digitalocean as your cloud provider? I find craigslist super slow when I use my own vpn server hosted at digitalocean. Maybe they are limiting digitalocean connections since lots of scrapers host their bots there?

jonpolak commented 8 years ago

@VikParuchuri CONFIRMED craigslist blocks EC2 and DigitalOcean IPs

WIRUT commented 8 years ago

@jonpolak I was able to test it with Calgary as well as using Amazon's EC2 services. I didn't bother using digital ocean. I'm not sure which settings you want. I did set SITE = "Calgary", and I think I bypassed the area and just hardcode the AREA in for this test instance.

Amazon EC2 was nothing different either.