search

NYULibraries / spatial_data_repository

NYU's GeoBlacklight instance @ geo.nyu.edu
https://geo.nyu.edu
6 stars 2 forks source link

Add GeoBlacklight suggested robots.txt entries #338

Closed spilth closed 2 months ago

spilth commented 2 months ago

Problem

We have a lot of traffic that is likely caused by crawlers and bots scraping the site.

Solution

Use robots.txt entries as suggested by the GeoBlacklight documentation to cull some of these requests.

Type

Enhancement