search

colonelpanic8 / okcupyd

A Library that enables programmatic interaction with okcupid.com, using okcupid.com's private okcupid JSON API and html scraping when necessary.
MIT License
110 stars 18 forks source link

OKC IP throttling issues #78

Closed kfred closed 8 years ago

kfred commented 8 years ago

I'm finding that I have been getting throttled. I have a hotspot shield, so once my scripts stop running, I disconnect then reconnect and activity continues; however, it only lasts for so long before it gets throttled again.

Confirmed it's IP- and not account-based, as I've successfully logged into and used the app on my phone over cellular connection while my account connected on my home network was throttled.

About 2 years ago I wrote a casper.js-based scraper, and I changed the headers to a Chrome or another browser, which appeared to have worked for me then.

Is there a way to change the browser header info send in the Web requests? Have you encountered throttling before and are there any best practices on how to work around the throttling?

Thanks!

colonelpanic8 commented 8 years ago

the browser header isnt the issue, i think youd just need to do some type of rate limiting to avoid this. We have something built in that handles this https://github.com/IvanMalison/okcupyd/blob/master/okcupyd/session.py#L53

kfred commented 8 years ago

Ah, cool, thanks. I'll check that out.

kfred commented 8 years ago

Cool, testing varying rate_limit's now. Nice feature.