Support for user-defined robots.txt definitions

[jonasstrehle]

see POC unyt-org/metadata-poc@main/common/metadata/resolver.ts

We should think of a good way how to add sitemap information to specific entrypoint route definitions