search

tedivm / fedimapper

An API for the Fediverse - The Software behind the Fediverse Almanac
https://www.fediversealmanac.com
MIT License
16 stars 2 forks source link

Instances that add robots.txt blocks should have their data removed #5

Closed tedivm closed 1 year ago

tedivm commented 1 year ago

This should only remove data retrieved from their server, while still allowing data from other servers (so that an instance with a lot of blocks can't hide that fact, while we still respect that instance's desire not to be crawled).