Documentation: Update `robots.txt` with new domain name

[amotl]

About

Related to the crate.io -> cratedb.com switchover...

• https://github.com/crate/crate-docs-theme/pull/439

... we discovered a few leftovers. All the robots.txt files, referring to the corresponding sitemap URLs, still use https://crate.io.

Sitemap: https://crate.io/docs/crate/crash/en/latest/site.xml

Examples

• https://github.com/crate/crate-python/blob/0.35.2/docs/_extra/robots.txt
• https://github.com/crate/crash/blob/0.30.2/docs/_extra/robots.txt

Solution

🏗️ Go through all relevant repositories [^1] and adjust that file. 🍰

[^1]: https://readthedocs.org/dashboard/ displays an exhaustive list.

/cc @michaelkremmel, @msbt

[msbt]

@amotl thanks for the find, but since robots.txt is only applied/scanned when in root / this is not an immediate issue. However, I'll take care of it asap.

[amotl]

Aha! It looks like https://cratedb.com/robots.txt and https://cratedb.com/sitemap.xml do not include any references to the resources at https://cratedb.com/docs/ at all? Do you know what may have caused this change?

[amotl]

Hi Matthias. Thanks for telling me about it. I just discovered this anomaly, but maybe my assessment was wrong. I guess the other ticket has a higher priority -- while not knowing about its origin, I just classified it as a bug. ;]

• GH-59

[msbt]

@amotl @matkuliak sorry for the PR spam, I think I've updated it everywhere. I found 2 bugs and updated the SQL-99 again to allow indexing. Those however seem to have failing builds:

https://github.com/crate/crate-pdo/actions/runs/7694636441 https://github.com/crate/crate-python/actions/runs/7694675724 https://github.com/crate/crate-dbal/actions/runs/7694875283 https://github.com/crate/crate-clients-tools/actions/runs/7694887315

[amotl]

Thank you. I will have a look at those RTD builds.

[amotl]

Thank you for updating all the robots.txt files so quickly. I think it will be safe to close the issue.