Add `robots.txt` and `sitemap.xml`

[AbrilRBS]

This PR aims to show how we could improve the discoverability of recipes in Conan Center by aiding in the crawling process of its contents. But we first need to understand what we are missing before going ahead with this PR, and get numbers on our current status to be able to measure the impact of these changes after we merge it. We also should check what options netx.js provides to improve the situation automatically. Are we not doing something we should?

Context

Even though next.js handles some of the SEO aspects by itself, and things like the server side rendering feature is great for it as explained in their docu, it does not automatically handle by itself the creation of either sitemap.xml nor robots.txt, files which can greatly help in ranking by grating search engines a full look into the site.

• For the sitemap.xml creation, I've followed this guide from the next.js docs. Note that as specified in the sitemap.xml Google documentation, sitemaps should not contain entries to subdomains, and we should let those handle it by themselves - In our case neither the docs (Which is a known issue, tracked in https://github.com/conan-io/docs/issues/3101 and https://github.com/conan-io/docs/issues/1691) nor the blog provide one, but that's a fight for another day
• For the creation of the robots.txt I've followed the path of least resistance as documented here in Google and here in next.js: Allow everything as it was done by default without it, but ensure a link to the sitemap.xml file is available. More info can be found here

Issues

• The main issue with this PR is the hardcoding of the conan.io domain in the sitemap and robots files, no idea if it can be avoided somehow, happy to get some pointers there.
• As discussed in the begining. have no insight into the current numbers, we have to get get them before moving ahead with the PR

Future steps

Once this hits the live site, and as we currently lack Search Console access, we can submit the new sitemap.xml to Google even if we don't have readily available access to the Search Console property, but we would need help in asking Google to recrawl the site, as that's only possible from within Search Console, for which access to the property is currently ongoing

cc @memsharded

[CLAassistant]

All committers have signed the CLA.