Improve Google indexing for documentation

[DinithiDiaz]

Description

We need to enhance the Google indexing of the MI documentation site to improve its visibility and search engine ranking. This issue will track the various tasks required to optimize the site for better indexing and search performance.

[DinithiDiaz]

We can use Google Search Console and PageSpeed Insights to track indexing progress and measure improvements in performance. Major tasks identified:

• Task 1: Submit an updated sitemap to Google Search Console.
• Task 2: Analyze and address any indexing errors reported in the Google Search Console.
• Task 3: Improve the site's loading speed and performance to enhance indexing efficiency. (Will be tracked via https://github.com/wso2/docs-mi/issues/861)

[DinithiDiaz]

Task 1: Submit an updated sitemap to Google Search Console

The sitemap for the MI docs is available at https://mi.docs.wso2.com/en/latest/sitemap.xml.

A sitemap is an essential file that helps search engines like Google understand the structure of the website and index it effectively. MkDocs generates a sitemap by default, and I verified that it’s correctly set up.

I have used the following tool to validate the sitemap:

https://www.xml-sitemaps.com/validate-xml-sitemap.html

---

Submitted the sitemap to the Google Search Console.

[DinithiDiaz]

Task 4: Add canonical tags to all EI documentation pages to refer to the latest version of MI documentation

Issue:

Currently, when searching for an MI-related query on Google, the search results often display pages from the EI documentation space instead of the latest MI docs. This occurs because the content in the older EI pages is similar to the MI documentation, and Google might be indexing those pages due to various reasons, such as better ranking history, age of the site, or lack of a clear canonical reference.

To address this, had an offline discussion with Thimuth and based on his suggestions, we need to follow the below approach to address this.

We need to explicitly inform Google which version is the latest using the canonical tag.

We need to ensure that Google is directed to the latest version of the MI documentation page by using the canonical tag. Since we use the /latest URL path in the latest MI docs space, this can be done straightforwardly for most pages.

Example:

<link rel="canonical" href="https://mi.docs.wso2.com/en/latest/get-started/introduction/" />

When there is no corresponding page in the latest MI documentation, the canonical tag should point to the relevant versioned documentation page of MI doc space.