Create proper `Robots.txt` and `sitemap.xml`

[p0lygun]

In the website logs, I see increased entries of "bots" so for that both Robots.txt and sitemap.xml should be generated,

Resources

• Robots.txt
  • https://ordinarycoders.com/blog/article/robots-text-file-django
• sitemap.xml
  • https://docs.djangoproject.com/en/4.2/ref/contrib/sitemaps/

Whether both the files must be configured in a new app called config or in the root project bfportal, needs to be discussed

[p0lygun]

Found that wagtail provides auto sitemap generation https://docs.wagtail.org/en/stable/reference/contrib/sitemaps.html