search

pittquantum / PittQuantumRepository

Flask backend for the Pitt Quantum Repository website
https://pqr.pitt.edu/
MIT License
16 stars 5 forks source link

Add Google, Bing, etc. sitemaps #52

Closed ghutchis closed 9 years ago

ghutchis commented 9 years ago

Adding Google, BING and other search engine sitemaps should improve indexing, e.g.

http://flask.pocoo.org/snippets/108/

ghutchis commented 9 years ago

When will this show up on pqr.pitt.edu? I'd like to submit to Google. Thanks.

JoshuaRogan commented 9 years ago

Should be up now.

ghutchis commented 9 years ago

Great, but what about 63000+ /mol/X pages? Thanks.

ghutchis commented 9 years ago

It's also not trivial, since each sitemap file can only have 50,000 locations in each. I'd suggest splitting the /mol/ URLs by first letter, e.g., all "A", etc. in separate sitemaps.

Break up a large sitemap into a set of smaller sitemaps to prevent your server from being overloaded by serving a large file to Google. A sitemap file can't contain more than 50,000 URLs and must be no larger than 50 MB uncompressed.

Use a sitemap index file to list all your sitemaps and submit this single file to Google rather than submitting individual sitemaps.

JoshuaRogan commented 9 years ago

I broke up the urls by the two letter convention we used in the folders.