robots.txt disallows all robots

4chan / 4chan-API

Documentation for 4chan's read-only JSON API.

http://www.4chan.org/

1.02k stars 73 forks source link

Closed spacehash closed 4 years ago

spacehash commented 4 years ago

Is this a misconfiguration?

desuwa commented 4 years ago

Nope.

spacehash commented 4 years ago

So, perhaps I'm misunderstanding robots.txt. Care to explain?

desuwa commented 4 years ago

It's mostly for search engines and those normally don't care about json APIs.