Exclude old lobid services in robots.txt from crawling

hbz / lobid

Linking Open Bibliographic Data

https://lobid.org/

Eclipse Public License 2.0

16 stars 4 forks source link

Closed acka47 closed 6 years ago

acka47 commented 6 years ago

I think we just have to make it:

User-agent: *
Disallow: /resource/
Disallow: /subject/
Disallow: /person/
Disallow: /organisation/

@fsteeg We should do this soon.

fsteeg commented 6 years ago

Deployed to production (stage has a different robots setup), see:

Please reopen if something is wrong.

ChristophEwertowski commented 6 years ago

acka47 commented 6 years ago