search

acl-org / acl-anthology

Data and software for building the ACL Anthology.
https://aclanthology.org
Apache License 2.0
423 stars 283 forks source link

Add .xml, .bib, and .endf files to sitemap #594

Open mjpost opened 4 years ago

akoehn commented 4 years ago

Should they be in search results?

mjpost commented 4 years ago

They currently are. If we want them not to be (which I think we might), we could add a "noindex" specification.

mjpost commented 4 years ago

Here is the coverage information from Google's search console:

image

Also, items that are excluded:

image
akoehn commented 4 years ago

I had a short look at the sitemap definition again and there does not seem to be a way to exclude files. That has to be done by a robots.txt or http headers (or we mark all the links as rel=nofollow).