Closed loganfranken closed 3 years ago
I just went ahead and migrated as-is: https://dev-webguide-ucsb-edu-v01.pantheonsite.io/content/seo
Thanks for the migration efforts with this SEO page. I adjusted it a little and think we can grow this out a lot more, but we have a good enough starting point in that we are addressing
Ready to close
384
Legacy Page https://webguide.ucsb.edu/seo/
Location in New Hierarchy ???
This is still relevant, although the "SEO" title might feel a little dated.
To help search engine crawlers properly index your website, include a robots.txt and sitemap.xml in the root directory of your website.
robots.txt
The robots.txt file is a simple text file that instructs the search engine how to crawl a website. More specifically, the robots.txt specifies which directories or files search engine crawlers should not index. For example:
Please note that robots.txt can only provide recommendations to search engine crawlers; crawlers may still ignore these instructions. Thus, do not use robots.txt to secure sensitive information. For more information on how to create a robots.txt file, read the Web Robots Pages.
sitemap.xml
A sitemap.xml can also be added to the root directory to provide search engine crawlers with a XML version of your site's structure. This sitemap can be referenced from your robots.txt:
For more information on creating a sitemap.xml, visit sitemaps.org.