Add robots.txt to packages.edgedb.com

edgedb / edgedb-pkg

EdgeDB Release Packaging Toolkit

1 stars 2 forks source link

Add robots.txt to packages.edgedb.com #40

Closed ambv closed 3 years ago

ambv commented 3 years ago

Currently a non-trivial number of downloads is Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://aspiegel.com/petalbot).

We need to let Web crawlers know they're off limits for our downloads pool.

ambv commented 3 years ago

Let's make sure package managers and other download tools we're seeing won't be affected by robots.txt denying downloads.

ambv commented 3 years ago

https://packages.edgedb.com/robots.txt