search

anyaudio / anyaudio-server

:musical_note: Simple server to search youtube and give direct audio download and stream links
91 stars 19 forks source link

Add robots.txt #137

Closed singhpratyush closed 7 years ago

singhpratyush commented 7 years ago

Add a /robots.txt endpoint.

Disallow /api/* and crawl delay of 10 secs.

singhpratyush commented 7 years ago

Googlebot is requesting /api/v1/d and /api/v1/stream_handler. Need to do this quickly.

singhpratyush commented 7 years ago

Cool!