MDAnalysis / MDAnalysis.github.io

MDAnalysis home page mdanalysis.org as GitHub pages.
https://mdanalysis.org
14 stars 39 forks source link

make sitemapindex valid: only include *same site* sitemaps #145

Open orbeckst opened 4 years ago

orbeckst commented 4 years ago

According to https://www.sitemaps.org/protocol.html#index

Note: A Sitemap index file can only specify Sitemaps that are found on the same site as the Sitemap index file. For example, http://www.yoursite.com/sitemap_index.xml can include Sitemaps on http://www.yoursite.com but not on http://www.example.com or http://yourhost.yoursite.com.

That means we cannot include docs.mdanalysis.org, userguide.mdanalysis.org with www.mdanalysis.org in the sitemapindex.xml.

Apparently, the algolia crawler does not care and still finds them (see https://github.com/algolia/docsearch-configs/issues/2103) but technically this is not legal and we should change it

orbeckst commented 4 years ago

I am not 100% sure if we need "proove ownership" for "cross submits" as detailed in https://www.sitemaps.org/protocol.html#location, namely our robots.txt file might need to explicitly list the sitemaps for doc and userguide. Or are these sitemaps ok because they are only accessed from the actual docs.mdanalysis.org/.... url?