search

4chan / 4chan-API

Documentation for 4chan's read-only JSON API.
http://www.4chan.org/
1.03k stars 74 forks source link

threads.json endpoint being blocked by CloudFlare's Browser Integrity Check #57

Open jiva opened 6 years ago

jiva commented 6 years ago

See title. I have a script processing the threads.json endpoint and I recently noticed that it's getting blocked by CloudFlare's browser integrity check. Can the check be disabled for that endpoint, since it's meant to be consumed by scripts?

desuwa commented 6 years ago

This will be fixed soon.

Regaccio commented 6 years ago

It seems like this issue is back. I'm not a developer, just a lowly script user, but I noticed the issue returned when using 4chan X. It seems like the script can't access a.4cdn.org due to 503 errors, and CSS is also broken unless I visit 4cdn.org directly and go through Cloudflare's Browser Integrity Check. Same issue as ccd0/4chan-x#1746

desuwa commented 6 years ago

This happens when you send too many requests too often. You'll get unblocked automatically, eventually.

Regaccio commented 6 years ago

I see. I had thread updating set to 1 second so that could be the problem, even though the limit on requests is one per second. So basically I just need to set the thread updater to only update every ten seconds as the API rules dictate and that should reduce the number of requests, thus preventing this from happening anymore?

Also, what exactly do you mean by too many requests too often? In too short of a time frame (e.g. a bunch of tabs open all making new requests quickly), or over a long period (e.g. going afk for hours and leaving browser open with a tab making regular quick requests)? Just want to know so I can avoid this in the future.

If there's any way to get unblocked manually that would be nice as well.

desuwa commented 6 years ago

The doc says no more than one request per second. In practice that limit is much higher and unless you have tons of threads constantly autorefreshing every second you should be fine.

chairus commented 5 years ago

@desuwa Given a scenario where I have N threads serving users and each thread would most likely run concurrently and doing 1 request per second, what would be the minimum number of N so that I wouldn't get rate limited?

Thanks

desuwa commented 5 years ago

Anything above 1rps can be subject to rate limiting.

chairus commented 5 years ago

Does that mean I am limited to only having 1 thread to service my users?

desuwa commented 5 years ago

It's not directly related to the number of threads you use. As long as your application doesn't make more than 1 request per second on average, you'll be fine. Again, that limit is higher than that in practice and is enforced in a pretty lenient manner, but there's no guarantee it'll always stay this way.