search

Multibit-Legacy / multibit-website-legacy

Website for multibit.org
Other
12 stars 25 forks source link

Add a robots.txt to stop crawling of binaries #234

Closed jim618 closed 9 years ago

jim618 commented 9 years ago

I think crawlers are downloading the out of date binaries we provide. This just eats bandwidth.

Perhaps a robots.txt file where we exclude the releases dir would lessen this

gary-rowe commented 9 years ago

We already have a robots.txt file, but it wasn't filled out very much. I've added more detail.

Ready for review and close.

jim618 commented 9 years ago

Should help our bandwidth a little. Closing