search

NeuromatchAcademy / mastodon

A glitchy but lovable microblogging server
https://glitch-soc.github.io/docs/
GNU Affero General Public License v3.0
7 stars 2 forks source link

Update robots.txt #45

Closed github-actions[bot] closed 6 months ago

github-actions[bot] commented 7 months ago

This PR was generated by the Update robots.txt action and contains updates to our robots.txt file, pulled from Dark Visitors.

sneakers-the-rat commented 7 months ago

Hmm why it removing some

sneakers-the-rat commented 7 months ago

IG they added a new category "undocumented AI agent"

hobgoblina commented 7 months ago

ah, wanna just ignore this for now then and update the script to include those?

sneakers-the-rat commented 7 months ago

Ya ya that sounds right.