SEO. Add robots.txt and Sitemap for improved website crawling

[daryakuritsyna]

Our website currently lacks a robots.txt file and a sitemap, which are essential components for directing and optimizing search engine crawling. The absence of robots.txt at http://activebridge.org/robots.txt results in a 404 error, and the non-existence of a sitemap makes it difficult for search engines to discover and index our content efficiently.

robots.txt file:

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/

Sitemap: https://activebridge.org/sitemap.xml

Include the path to the sitemap within the robots.txt file to assist search engines in locating and crawling our content. The sitemap location should be the absolute URL to where sitemap.xml file will reside.

XML sitemap Create an XML sitemap that lists all the important URLs for the website. This should include pages that we want search engines to index. Also, could we design the script line to update Sitemap regularly, especially after adding new content f ex. articles and so on?

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
    <url>
        <loc>https://activebridge.org/</loc>
        <lastmod>2024-04-01</lastmod>
        <priority>1.0</priority>
    </url>
    <url>
        <loc>https://activebridge.org/about</loc>
        <lastmod>2024-03-20</lastmod>
        <priority>0.8</priority>
    </url>
    <url>
        <loc>https://activebridge.org/about/team</loc>
        <lastmod>2024-03-20</lastmod>
        <priority>0.7</priority>
    </url>
    <url>
        <loc>https://activebridge.org/services</loc>
        <lastmod>2024-03-15</lastmod>
        <priority>0.8</priority>
    </url>
    <url>
        <loc>https://activebridge.org/blog/</loc>
        <lastmod>2024-04-01</lastmod>
        <changefreq>weekly</changefreq>
        <priority>0.9</priority>
    </url>
    <!-- Additional URLs here -->
</urlset>

Upload Files to Server:

• Place the robots.txt file in the root directory of the website (e.g., http://activebridge.org/robots.txt).

• Upload the sitemap.xml file to the server and ensure it's accessible via http://activebridge.org/sitemap.xml.

[Smoke-ck]

173

[Smoke-ck]

Hi @daryakuritsyna, it's already in prod.

[Smoke-ck]

174