Create robots.txt for wiki

heidelberg-makerspace / dai-makerspace-do-something

Issue tracker for tasks you could do in the DAI Makerspace

5 stars 0 forks source link

Create robots.txt for wiki #86

Closed emka closed 7 years ago

emka commented 7 years ago

Currently, web crawler bots are following every possible wiki link (for example I saw a log entry for reading the calendar in 1982).

Create a robots.txt that makes sense, i.e. allow actual wiki content and disallow special pages like creating accounts or calendar entries far away from the current date.

emka commented 7 years ago

I created a basic robots.txt based on the Mediawiki manual. I am not sure if that is sufficient, please add your thoughts and suggestions here.

emka commented 7 years ago

I added load.php and api.php today.

emka commented 7 years ago

Looks fine now...