Remove restrictions from robots.txt when deploying to kitspace.org domain

kitspace / kitspace-v2

Work in progress on using Gitea as the Kitspace backend.

GNU Affero General Public License v3.0

13 stars 6 forks source link

Open kasbah opened 2 years ago

kasbah commented 2 years ago

Just a reminder to not to use the restricted robot.txt from staging.

AbdulrhmnGhanem commented 2 years ago

I think we should block the spamming bots too. Maybe we should use https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker

kasbah commented 2 years ago

I'm not sure we should be spending time on that right now.