Migrate "SEO" Page

[loganfranken]

384

Legacy Page https://webguide.ucsb.edu/seo/

Location in New Hierarchy ???

This is still relevant, although the "SEO" title might feel a little dated.

---

To help search engine crawlers properly index your website, include a robots.txt and sitemap.xml in the root directory of your website.

robots.txt

The robots.txt file is a simple text file that instructs the search engine how to crawl a website. More specifically, the robots.txt specifies which directories or files search engine crawlers should not index. For example:

User-agent: *
Disallow: /images
Disallow: /scripts
Disallow: /style

Please note that robots.txt can only provide recommendations to search engine crawlers; crawlers may still ignore these instructions. Thus, do not use robots.txt to secure sensitive information. For more information on how to create a robots.txt file, read the Web Robots Pages.

sitemap.xml

A sitemap.xml can also be added to the root directory to provide search engine crawlers with a XML version of your site's structure. This sitemap can be referenced from your robots.txt:

User-agent: *
Disallow: /images
Disallow: /scripts
Disallow: /style
Sitemap: http://www.example.ucsb.edu/sitemap.xml

For more information on creating a sitemap.xml, visit sitemaps.org.

[loganfranken]

I just went ahead and migrated as-is: https://dev-webguide-ucsb-edu-v01.pantheonsite.io/content/seo

[tenken]

• thanks Logan! I will reach out to Ryan to fix where this sits in the site menu navigation, it appears as though either the Content menu item or this page isn't landing within a menu item at this moment.

[rvizena]

Thanks for the migration efforts with this SEO page. I adjusted it a little and think we can grow this out a lot more, but we have a good enough starting point in that we are addressing

• robots.txt
• sitemap.xml

Ready to close