When released remove robots.txt file

openaustralia / yinyo

A wonderfully simple API driven service to reliably execute many long running scrapers in a super scaleable way

https://yinyo.io

Apache License 2.0

6 stars 1 forks source link

Closed mlandauer closed 4 years ago

mlandauer commented 4 years ago

It currently is stopping any bot from scraping it so news doesn't by accident get out before we're ready.