feature: XML sitemap and robots.txt

[jbmoelker]

Changes

• Generates XML sitemaps for all pages, based on getStaticPaths() and using the official @astro/sitemap package.
• Adds a basic robots.txt file.
• Adds a reference to the generated XML sitemap in the robots.txt with the deployment's site URL.

Associated issue

Resolves #5

How to test

1. Open preview link
2. Navigate to /sitemap-index.xml
3. Verify the sitemap is generated and correctly links to sub sitemaps
4. Navigate to the sub sitemap from the sitemap index (/sitemap-0.xml)
5. Verify it correctly lists all available pages (check that one exists for example)
6. Navigate to /robots.txt
7. Verify it correctly has a rule pointing to the sitemap index
8. Navigate to any HTML page on the website
9. Verify that it has a link[rel="sitemap"] correctly pointing to the sitemap index

Checklist

• [x] I have performed a self-review of my own code
• [x] I have made sure that my PR is easy to review (not too big, includes comments)
• I have added/updated tests to prove that my feature works (if not possible please explain why)
• I have made changes to the README and if the change affects the project setup (npm commands changed, new service added, environmental variable added)
• I have added a decision log entry if the change affects the architecture or changes a significant technology
• [x] I have notified a reviewer

[cloudflare-pages[bot]]

Deploying with    Cloudflare Pages

Latest commit:	f7ab6be
Status:	✅  Deploy successful!
Preview URL:	https://2df41e43.head-start.pages.dev
Branch Preview URL:	https://feat-robots-txt-and-sitemap.head-start.pages.dev

View logs

[jbmoelker]

Seems like robots.txt.ts is engineered only to be able to reuse the site url. You can make that part of the readme ("make sure you configure a robots.txt file"); I'd say it's not important for a dev environment to have a working sitemap (ie. having the localhost:2343 URL in there). Alternatively I would recommend an Astro integration that writes a robots.txt (post build, perhaps, but it shouldn't matter much). I think an integration is a more logical/expected approach and is easier to extend.

Thanks @phortuin!

Yes, I was in doubt about which route to take. Had a different approach before. The annoying thing about hardcoding things is that if Head Start is used as a template for another project and that project changes hardcoded values, it makes it harder to sync with the upstream repository.

And I think we'll need this somefile.ext.ts trick more often, for instance for an opensearch.xml.ts with more dynamic values.